Rust Programming Language RustFest Global 2020, 23 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: glowcoil - Ochre: Highly portable GPU-accelerated vector graphics — RustFest Global 2020

Description

As screen resolutions and refresh rates increase, user interfaces increasingly need to make use of GPU acceleration in order to achieve good performance and battery life. However, the wide range of existing GPU feature sets and APIs makes it difficult to do so in a portable manner. Ochre is a Rust library for vector graphics and text rendering which is structured to take advantage of the performance strengths of GPUs while remaining portable to diverse situations, including older hardware, laptops, mobile devices, and WebGL.

More at https://rustfest.global/session/23-ochre-highly-portable-gpu-accelerated-vector-graphics

A

Glow coil shows how vectors can act to create a great ui. In fact, they are easy to do on a slow gpu and they won't fall together when stacked.

B

Hi I'm micah johnston. I also go by the username glowcoil and today I'm going to be presenting a project that I've been working on called ochre, which is a gpu accelerated, vector, graphics and text, renderer um library for rust, and the primary use case that I've intended ochre for is ui rendering.

B

So, first I'm going to try to answer the question. Why am I making a vector graphics render in the first place? So I would make the claim that vector, graphics is by far the dominant type of representation for graphical content in user interfaces.

B

Today we use it for our font formats and as well as emoji and html and css are they basically comprise a vector format and a pretty massive amount of uis use, html css and the browser rendering engine today so and then beyond that most os platform, ui toolkits and cross-platform ui toolkits all use pretty much use vector graphics at this point and that's for some important reasons. So, first of all, it's just more space efficient to store the vector form of something than the image form, especially at high resolutions.

B

But beyond that, it's a resolution, independent format, so suppose you want your app to run on both a retina macbook and an older 1080p monitor if you're using images you have to export a new image for the retina format, whereas vector graphics can be rendered the same on different pixel densities.

B

So that's a pretty powerful benefit and then beyond that, it's just a good toolkit for if you have anything, that's layout, dependent based on the size of your window or, if you're doing something like rendering a a waveform visualizer in a synth or a a line graph in a data visualization program, vector graphics, is just it's a really good tool kit for for procedural visualizations like that, so we would like to have a vector graphics, renderer for our uis.

B

But um now let me try to answer the question: why does it need to be gpu accelerated, so there are kind of two trends over the past 15 to 20 years or so that um are the reason I would say: gpu acceleration is important for you guys so the first one of these is that resolutions of our screens are going up.

B

This is a visualization of iphone screen sizes over time, so there's a pretty drastic increase and this doesn't even include the latest iphone which, if I remember right, it has something like 2700 by 1100 pixels. So that is a lot of pixels we need to.

B

We need to render every frame if you want your app to run at a smooth, 60 frames per second, you have to render a lot more pixels every second, and that means using more computing power so, but at the same time we don't want to drain the battery if we're on a laptop or a mobile phone, and you need to hit that 60 frames per second deadline if you want to have a smooth app. So um the other important trend is that gpus have become really ubiquitous in consumer hardware.

B

So you can. You can get a lot more computation done both per second and per watt, with a gpu than a consumer single cpu core. So this lets you be both more efficient and hit your frame deadlines and also more power efficient. So you don't drain the battery or use too much power just for rendering something simple like a ui which, because presumably you'd like to use the rest of your cpu for other applications. Your app is presumably not just a ui.

B

So um because of increasing resolutions- and I should note- refresh rates- are also starting to become an issue because 120 hertz 144hz monitors are starting to enter the market. So that's even more pixels you have to paint per second um gpus are really good for highly parallel tasks and rendering happens to a lot of aspects of 2d rendering happen to be highly parallel.

B

Since a lot of tasks, you simply do the same operation per pixel, so it's a good fit for gpus, and the set of gpus that are available in consumer hardware are a good fit for returning uis. So and as kind of evidence of this, both mac os and windows have been using gpu hardware to accelerate the basically the step of taking all of the different windows. You have open and painting them all onto one one frame to display on your monitor the, which is an operation called compositing.

B

Both mac and windows have been using the gpu to do that for around 15 years. I think mac since around 2004 and windows, since 2007 with vista so and in addition to that browsers, are also increasingly taking advantage of the gpu. So um there's evidence that it is a good idea to use gpu acceleration to render the ui for both efficiency and power efficiency reasons.

B

So hopefully, I've convinced you that a gpu accelerated vector renderer is a desirable thing to have for a ui, but using the gpu comes with a catch, and that's basically that um you can't just write a single program and then run it on every gpu out in the wild there's a big variety of manufacturers and then of apis and platforms which give you some subset of the apis, but not all of them.

B

So I have this table here showing which apis are available on which operating systems and there isn't really an api that has full coverage of all the operating systems. You might want to target opengl looks promising, but it's it's officially deprecated on apple platforms and even before that there was only an older version with more limited features was supported, and um this table actually looks a bit more rosy than the real situation, because vulcan and the newer versions of directx and metal are only supported on newer hardware.

B

So suppose you have, you have a slightly older phone or you have a 10 year old, desktop computer. It may not be able to run the newer features, so even if you're, if you've picked an operating system and a a gpu api, you have to negotiate which feature level you're willing to target and which users you're willing to exclude from your application.

B

So portability is kind of a hard question. You have to figure out how you're going to make your application use different apis on different platforms.

B

If you do want to be cross-platform, probably the hardest part of this is that when you write code that runs on the gpu, you use what's called a shading language and each of those different apis have different shading languages, so either you're going to have to rewrite your shaders for each platform or you're, going to have to figure out some cross compilation setup where your build system includes a shader compiler, which increases complexity, and even then you still have to negotiate platform-specific features when you're doing so.

B

So it's kind of a big headache and the approach that I've taken with ocr is just to choose a very small subset. The smallest possible subset of gpu features that are going to be available pretty much anywhere that are still going to. Let us leverage, gpu performance advantages and are still going to. Let us do the ui graphics that we would like to do so um to get into a little bit more detail about what a vector render actually has to do.

B

There are kind of two aspects, so the first aspect is you take these shapes so, for instance, a font glyph a letter in a font.

B

It's basically defines a solid region bounded by a curve, and the renderer has to determine which pixels are inside or outside that curve and then fill them in with the appropriate color, whether that's a solid color or a gradient, and then the other aspect is taking multiple shapes like that and compositing them in order using, what's called the painters algorithm, where it's called that, because later things that you draw paint over earlier things, so the second step compositing, which is on the left here, gpus, are really good at it and, like I mentioned before, this is what operating systems and browsers have been making a lot of use of gpus for for a long time now.

B

So it's it's just an easy thing to do with gpus and they're very good at it. It's much more power efficient than doing it with the cpu.

B

On the other hand, the painting, the operation that I will call painting, which is determining inside and outside of a shape like this is it doesn't come naturally to gpus, since they kind of only natively speak in triangles, so you have to somehow translate these types of shapes into triangles in one way or another for the gpu to understand. So that's the hard part. There are a lot of different approaches to doing so. So I have this kind of spectrum here from renders that use more cpu to render as they use more gpu.

B

This is a it's a huge oversimplification, so this is super subjective and on in different ways. You could argue that these should be in different orders, so this is just intended to kind of give a broad overview. So on one end we have doing rendering entirely on the cpu and then, as we move down, there's tessellation, which is an approach that basically busts apart a shape like this into triangles and then just shovels, the triangles over to the gpu to be rendered, and it's robust. It's simple.

B

It does work well for performance, but there are some big downsides, namely that gpus aren't capable of rendering triangles with the type of anti-aliasing that you need for small text.

B

So you kind of have to do a hybrid approach if you use tessellation um and maybe render your text on the cpu and your bigger shapes on the gpu and um there's another approach called stencil and cover- and it's it's called that because it uses a feature of the gpu called the stencil buffer to, rather than doing the kind of mathematically hard operation of breaking apart, a concave curved shape into triangles it.

B

Just it draws sort of depicts a point and draws a fan of triangles out from that point in such a way that they all cancel each other out to leave only the points inside the shape filled and the points outside the shape empty, and it has the same disadvantages as tessellation, where the it's hard to get really high quality anti-aliasing and um while it makes more use of the gpu, it's also an inherently less efficient algorithm than cpu rasterization methods. So it's kind of a trade-off. It's not a it's!

B

Not a pure win, um there's an example of a library that does this called nano vg, which it uses the stencil and cover approach. It has to take a hybrid approach with text, rendering where it renders text on the cpu and big shapes on the gpu.

B

Nano vg was a big inspiration to ocr, because it's it's the same kind of minimal library focused on portability and working on as many situations as possible.

B

So as we move further down, we get these more complicated approaches um such as pathfinder, which is a rust library written by patrick walton and pathfinder, was probably the number one inspiration for ochre, I'm not sure if I would have have written ocr without pathfinder.

B

So I have to give some big acknowledgments to patrick walton, for that pathfinder is kind of a refinement of the stencil and cover approach that does a lot of cpu work to split the shapes up, so that you only have to do work near the edges rather than in the big, opaque centers of the shapes.

B

um So it's it's really a cpu gpu hybrid you're, doing work in both places, and it can. It can really outperform cpu rasterizers like cairo, but um it's it's kind of a hybrid approach like that. It's not pure gpu, and then you you.

B

um The next item on my list here is a vector, textures architecture, which is it works kind of differently there.

B

The way it works is you have a cpu preprocess and then you can render it many times on the gpu from different angles, for instance in a 3d scene, so it offloads a lot of work to the gpu, but unfortunately, it kind of has performance, tradeoffs more appropriate to a game than to a ui, because um the reason for that is the end-to-end render time from loading a font or or generating a scene from scratch to processing it uploading and rendering on the gpu is not even really faster than maybe full-on cpu rasterization.

B

Sometimes um so I I don't consider this a good approach for uis, but it is a good approach for other situations such as games and then, finally, last on the list, we have a pure gpu compute renderer, which basically uploads the scene as a data structure to the gpu uses. Modern gpu compute features to render it from scratch.

B

There um you'll notice pathfinder also appears here, because patrick walton in in recent months developed another renderer for pathfinder that uses gpu compute and, um along with that, there's another rust library called p gpu that um was developed by ray flavine and both of these get really impressive performance when you're using a a high-end gpu, they scale up, unlike any of the other approaches, to really using the gpu, um but there's there's kind of this central trade-off here, as you use the gpu better, you can scale up better with bigger gpus, but it makes it harder to work on older hardware and it makes it harder to port between different apis, and this is, as I understand it.

B

This is why pathfinder has both renders because the the non-compute render is um trying to achieve more portability.

B

So ocr is also an attempt to kind of strike a balance on this trade-off, and it is an attempt to strike a balance closer to tessellation in the portability sense, but kind of still achieve high enough anti-aliasing quality that you can render text with it. So um let me go over a little bit of my earlier process that led me to the current state of ochre.

B

So the first thing that I worked on, um I was working on this for about a year um starting about a year and a half ago, and it works like the vector textures architecture, as I mentioned earlier.

B

um So it has the trade-offs that I mentioned, where it's better suited to a 3d game where you render something that's the same many times from different angles, um and I I put a lot of work into this and then I decided it wasn't the appropriate approach for uis, so I've kind of shelved it for now, but um gouache will return eventually, um but I was searching for something that was that struck that trade-off better for you guys- and um this was the first thing I got really excited about- so I call it sparse, scanline rendering and the way it works is um it only renders the pixels that are intersected by the the outlines um and then it and then it uploads those two as horizontal lines to the gpu.

B

So I also like to call it the gl lines, gopher architecture.

B

um So you can see on this diagram anything, that's not a pixel intersected by the curve doesn't have to be processed on the cpu um and it just gets filled in by the gpu, which is good at good at doing kind of simple, highly parallel tasks like filling in a bunch of solid pixels.

B

So um so yeah you can see here. This is what the horizontal lines look like. They get uploaded to the gpu, and you can you can actually think of this architecture as run length encoding the image.

B

So you you have to upload less data and compute less data in the first place, because you inherently skip all the work of the solid spans, and this was um this had much better performance than I expected from how kind of weird of a design and how also how simple it is so for for complex scenes like the the there's a tiger scene, that this is a this is a clipping of.

B

I was getting times that were 10 times as fast as cairo, which is a single threaded cpu on the renderer, so that was really promising, but it has some downsides.

B

Basically, it doesn't handle um humongous, solid spaces very well, so, if you're rendering a full screen rectangle, the gpu gets stressed out by how many lines you're trying to shovel through it, and it is about five times slower than just doing it with two rectangles. So I I tweaked this approach, and I ended up closer to this- is actually very similar to what pathfinder does, but it basically does more on the cpu, whereas pathfinder does more on the gpu.

B

Basically, I break the shape down into edge edge tiles and spans, so rather than edge pixels in it and solid spans. I have eight by eight edge tiles and then solid 8 by n spans in the middle, and then I pack, these into an atlas texture upload that to the gpu and render it all using using triangles to make up those rectangles.

B

This is an example. Texture atlas, for this is what it looks like for the tiger. This is all of those little 8x8 chunks put in order in the atlas.

B

So so that's how ochre works and um I'm going to get a little bit more into how it how it works from an api design standpoint. So basically, I I wanted ochre to be usable as a component rather than kind of taking over the design of your program.

B

If you use it so when when it builds these tiles and spans the what it does is it just builds that data for you and then you can take that data and you can upload it yourself to the gpu so that lets you, um whether you're on directx or opengl or metal, whether you have an existing game engine you want to use or you're using some proprietary console api that I couldn't have even foreseen or added to ocr. As as an api. You can just add the support yourself very straightforwardly using some simple operations and.

B

And it will still use the gpu efficiently, but all the work, um all the work to build. This data is done for you by ocr and then so I like to I like to think of this.

B

As humble library design, where you you respect what the user wants to do, the user knows best their platform and performance things, performance constraints, and um you just make a library that that can serve as a component alongside the other components of an application, rather than kind of trying to take over and insist on things being done a certain way.

B

I think that's the best way to be portable because it allows many different situations to make use of your library- um and I guess one more shout out for inspiration here- would be the deer m gui library, which is a a ui rendering library for c plus plus, which takes a very similar approach. And it's been. It's been used in very diverse scenarios, including the the um code base for the large hadron collider. I believe so um being humble like this gets you a long way anyway.

B

So that's how ochre works and that's why I made it the way I made it um it's on github, I'm hoping to release it on crates that I owe fairly soon anyway. Thank you for.

B