Rust Programming Language RustFest Barcelona 2019, 27 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustFest Barcelona - Niklas Reppel: Amp Up Web Audio Applications with Rust and WebAssembly

Description

With the advent of the Web Audio API, creating browser-based audio applications became reality, potentially making creative applications available to a wider audience. There’s just one problem: writing real-time digital sound processing code in a garbage-collected language isn’t all that easy. A systems programming language like Rust, on the other hand, is perfectly suited for that task. Cue WebAssembly! Thanks to Rust’s convenient tooling, efficient DSP applications for the web browser can now easily be created, helping to make creative tools available on every platform.

https://barcelona.rustfest.eu/sessions/amp-up-web-audio

A

Niklas: hello?.

B

Good morning.

C

Thanks for having me here., I'm niklas, working for the local research and development centre of catalonia, eurecat,.

C

And my job is making audio software, nowadays still in c++, and I'm fairly new to rust. But.

C

Very well related to claudia saxer's talk yesterday.

C

My journey was pretty similar in the first few months.

C

Today, I'm going to talk about the web audio and how rust can be used for this, but also.

C

A little bit why rust might be a good option for audio software in general.

C

So why would you want to put complex web audio, or complex audio applications on the web in.

C

The first place?

C

I'm part of a community that does music with code..

C

I don't know who does not know, live coding, here?.

C

Everybody knows it,, I'm surprised.

C

Small programming languages, with domain-specific languages, and one of the problems that this.

C

Community has is that workshops often end up being installation workshops and after.

C

Two or three hours people manage to their things, which is kind of sad, because the stuff.

C

Is fairly easy to use and it's a lot of fun to use and might be educational, a lot of.

C

People might get interested in the programming, but if you need two days just to install,.

C

I don't know, whatever weird environment people develop the languages on,, it might discourage.

C

So if you can put it on the web, you just need to load a website,. You don't need a lot of.

C

Computer literacy and it may just lower the boundary for people to use your stuff and.

C

Might make it more fun for more people.

C

I went on a journey to try to put my own stuff to the web and I'm still fairly early in that.

C

Process, but I was looking for the perfect web audio stack,. So what are the requirements.

C

For audio software in general?.

C

You need a deterministic way of working for the audio thread, itself.

C

That means you, don't you shouldn't allocate memory in the audio thread.

C

You shouldn't lock,, you shouldn't wait on conditional variables and all those kind of.

C

Things,- and it is inherently multi-threaded because you always have the audio,, the realtime.

C

Thread and some control thread at least to either set parameters or schedule, stuff, sequencing.

C

Stuff, and so on, and so on.

C

Out of personal preference, what would be nice to have not only for the within but for audio.

C

In general, memory, safety is a thing because you suddenly get random memory. Contents pumped.

C

Directly into your ear,, what at work we affectionately call noise from hell,, then memory safety becomes.

C

Very important suddenly, just to be able to keep on working and not being deaf.

C

Portability is a thing because for me,, if at some point I would not be happy with the.

C

Web performance anymore and want to have a native programme,, it would be nice if it would.

C

Be easy to port. small footprint., I don't like bloaty software, and the easier it is to use, the better.

C

So options here, eked go with pure javascript, but it's not that portable,. I'm not an expert.

C

In javascript,, but I think compiling it to a native executable is something different.

C

It's a dynamic, scripting language.. That means it has probably something like garbage collection which, in some languages.

C

Can easily stop the audio for a second,, you get glitches, and it gets easily bloaty.

C

I don't want to chime into javascript bashing here, I like javascript,, but I don't think.

C

It's the best for this kind of purpose..

C

You could just compile c++ with emscripten, but that's neither very convenient or very.

C

Comfortable,, I don't know.

C

So I came across rust and webassembly and that seemed to be a fairly good option.

C

Why? well,, so, first of all I looked around and tried to find something that has already been.

C

Done and I came across this little synthesiser- that's not made by me- but by some person from.

C

Japan- and that was the proof of concept that showed me: okay,. It's actually working.

C

It's like a little tv303 simulation, kind of funny.

D

Even though it's still lacking sample playback.

C

And precise timing and all things that need to be done,. But apart from that everything.

C

That's written about it is in japanese,, which is a beautiful language, of which I have very.

C

Little command unfortunately,, so I need to put some effort into that.

C

So yes,, why use rust for audio applications in the first place?.

C

And for web audio applications in particular?.

C

So the memory aspect, memory safety aspect- I already lost a few words about that,. Why.

C

You would want to have that in audio software.

C

Rust boasts itself of fearless concurrency so for multi-threading applications. It might.

C

Actually be pretty good., it's a deterministic systems, programming, language.

C

Gives you stable results., you don't have any garbage collection to deal with and don't have to expect glitches.

C

You can port it to the web or, if you are not happy with the web audio software anymore,.

C

You could easily compile it to a native executable.

C

The webassembly integration is very seamless, compared to c++ or other languages where you.

C

Have to struggle with emscripten or whatever.

C

For rust, you just set a different target and there you go.

C

And last but not least, the performance.. I only did some very casual profiling in the developer console, but looking at the numbers.

C

The main audio processing method needed roughly 15, 20% less than the native javascript applications.

C

To test sample playback and to work around, in general, get used to working in rust and.

C

Webassembly. this usually needs headphones, so it doesn't do a lot on a single speaker, but it's a very.

C

Simple sample player and for profiling, the stuff I did the same thing in pure javascript.

C

And it seems to be more reliable with rust and webassembly.

C

So what are the ingredients of an application made with rust?.

C

First of all, you need a module compiled to a webassembly module.

C

Bindgen doesn't really work that well in that context,. So it's a raw library that you compile.

C

To webassembly., you need the audio worklet that is part of the web api that allows you to do custom processes.

C

That can use a webassembly model.

C

You need some setup and control script,. Obviously the main thread,, the user interaction.

C

If you want to do any scheduling,, any time-based stuff,, you need some additional webworker.

C

Thread for sequencing and these kind of things, and ideally some pure rust, dsp crates that.

C

Easily comply to webassembly, as opposed to bindings, like if you want to do some advanced.

C

Transform libraries like fftw or whatever,- it's probably harder,, but with the pure rust.

C

Crate that works pretty well.

C

What are the challenges of making a web audio application with rust?.

C

The audio, worklet and webworker stuff is a little bit tricky because you cannot load.

C

The models directly.

C

You have to fetch them in the main thread and then post them using the message. System,.

C

But once you've got through that, once you've got the setup done, it's fairly easy.

C

Setting up the shared memory between the webassembly and the main javascript is a little bit tricky,.

C

And as I already said,, it was the bindgen for the worklet part itself,. Exactly because.

C

Of the limitations of the audio worklet that you cannot load the module directly in the.

C

Work that you have to post it there in binary data.

C

What's already there?.

C

Crates that support, or that help you creating audio software with rust?.

C

There's chfft, a pure rust library. If you want to do convolutions.

C

I use that followed by normalisation of this part,, like for the simulation of the 3d impression,.

C

And this, if you want to check this out with headphones and play around with it, a bit,.

C

The current version is still time-based convolution, but the new version that I might upload soon.

C

Is based on actual fast convolutions.

C

There's an alternative called rust fft, but I haven't had a lot of luck with that.

C

Basic.Dsp crate, which I wasn't very happy with,, it's a very vector-based way of thinking.

C

And it's a little bit too detailed, maybe but anyway, it exists. So you might want to check.

C

It out, if you are into dsp and want to do some audio software with rust.

C

What's new york working all that well yet?,.

C

I haven't managed to do any threading directly from rust,, so in the sense that I want to.

C

Spawn a thread to do scheduling, tasks, so scheduling, music events,. So you still have.

C

To rely on the webworkers, I think,, unless somebody knows something more, and I'm willing.

C

To learn because, as I said, I'm still fairly new to that in the first place.

C

Debugging is a bit tricky,, as we already learned in the talk yesterday.

C

You should test your dsp code well to make sure it runs because to debug it directly.

C

In the webassembly part is fairly annoying.

C

and unfortunately, audio worklet is not yet available in firefox.

C

So far, it only works in chrome and opera,, which is a bit sad and I hope it will be implemented.

C

In a proper way in the future in firefox as well,, but we are not there yet.

C

So what is to be done??

C

One of my future tasks in general is to do some general benchmarking, because I want to.

C

Know how the rust software on the web, in particular, but also in general, compares to my c++ stuff.

C

Which I haven't done yet,, especially regarding fast fourier, transforms, which is a central.

C

Thing in my daily work.

C

to do past convolutions crate, to be able to do re-works on the web and other things,.

C

And in general I had hoped, maybe somebody would be inspired to do audio software and.

C

We can sharpen rust's profile as an audio language in general,, so I can avoid in the.

C

Future having random memory contents pumped into my ear, directly.

C

As I already mentioned,, that's, not a very pleasant experience.

C

So yes, some resources, there's a rust audio. Forum,. This discourse group.-.

C

The rust audio environment is actually bigger than you might think.

C

There is a vst crate that you can use to do plugins for audio work, stations,, which I haven't.

C

Used so far, but I know it exists,, so if you want to look for any resources,, that's where.

C

You would go.

C

this is for some examples,. If you want to learn from that,, especially with the webassembly.

C

Part. there is this original seminal example that I already presented.

C

It's all in japanese,, if you speak, japanese,, there's stuff written about that.

C

If not,, I did the sample player.

C

It's well commented, so it might be a nice starting point for your own applications. If.

C

You want to check that. and last, but not least,. A more complex example that I made and, as you say in my native language,.

C

Demo of that, what life coding means and what.

C

You can do with rust, audio so put the microphone away, and I will give you a little impression.

C

D

Thank you for listening.

D

I don't know if there's time for questions, or anything else?.

C

All of this is running in a web browser..

C

It doesn't need any server backend,, it just needs a web page and there you go.

E

Hi,, an amazing talk and demonstration,. Thank you very much..

E

Last time I checked for web audio in firefox, I think there was a limitation of 256 milliseconds.

E

On latency., I don't know how it is now but,, depending on which kind of application, latency is important.

E

Also, what is the difference between audio worklet and web audio?.

E

Niklas:, I would assume it's getting less and less.

C

250 milliseconds sounds quite a lot.

C

I must admit: I'm not. My own work is mostly based just on output,, not on input.

C

I don't do live processing where this would be very important,, but I've seen applications.

C

That do guitar effects on a web page and they are fast enough so that you can play it with.

C

The recent advances.

C

I don't know if I could find it, but there are guitar effects that run on the browser.

C

That are at least fast enough to test them and not completely bring you out of the rhythm.

C

When you bring guitar,, so I assume that would be better now.

C

Hi,, a great talk by the way.

E

Niklas: thank you.

B

In the ruffbox play-through there was a sort of scripting language that you typed.

B

And it made the music.

C

Niklas: yes., what is it? niklas:, that is just a very,, very simple, pattern,, basically, a sequencer that I made to test.

C

It., it doesn't have big functionality..

C

That's my next project basically, to make an actual more complex language for the synthesiser.

C

To use it more productively, or to port,, I don't know, my existing stuff to these kind.

C

Of things, and maybe other stuff. there's a lot of nice live coding languages,. So that's a future project.

B

C

Niklas: thank you.

E

Final question.

D

Raise your hand.

D

I'm curious if it would be technically possible to do this kind of audio processing.

A

With the main web audio without audio worklet, and if you can explain why this needs audio.

A

Worklet, maybe it would help motivate implementation.

A

Niklas:, there's always multiple ways to do: things, but yes,. One of the reasons for me.

C

Is that, for example, if the performance isn't enough,, I can use it natively as well, and.

C

In different contexts,, which might not be that great with pure javascript.

C

Of course,, everything of that could be done in pure javascript or with different text.

C

Apps, but I think the web api itself and the nodes it currently provides,, it's not that.

C

Flexible. for example, all the timing stuff you saw here is done in the rust part.

C

The precise time,, the scheduler,, the pattern language only gives out very coarse events.

C

And they are requeued and re-ordered in the rust, part, and so on, and so on,. So I think.

C

The complexity might just be easier to handle if you have a systems, programming, language.

C

There, instead of the basic web audio api facilities,, but of course that's all personal preference,.

C

So there are, of course, different ways to do: it, and whatever suits your needs,. You can.

C

E

Thank you, niklas., give him a round of applause.