Rust Programming Language Rust Linz, 23 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rust Linz, April 2021 - Jan-Erik Rediger - Leveraging Rust to build cross-platform libraries

Description

At Mozilla, Firefox is not the only product we ship. Many others — including a variety of smartphone applications, and certainly not just web browsers — are built by various teams across the organization. These applications are composed of a multitude of libraries that, when possible, are reused across platforms.

In the past 2 years we used Rust to rebuild one of these libraries: the library powering the telemetry in our mobile applications is now integrated into Android and iOS applications and since earlier this year on Desktop as well.

About Jan-Erik Rediger
At Mozilla, Jan-Erik writes stuff so others can measure thingies. He's an active member of the Rust community team since 2014. He coached at Rust workshops & runs Rust conferences in Europe. To get away from computers he picked up scuba diving to see the world underwater.

Rust Linz at https://rust-linz.at
Twitter: https://twitter.com/rustlinz
Jan-Erik: https://twitter.com/badboy_

Speak at Rust Linz: https://sessionize.com/rust-linz

A

Thank you. It's still, weird speaking to your own computer and not seeing anyone so uh welcome everybody. I'm very happy to be here and I'm here to talk about how my team at mozilla is using rus to build crossbot from mobile libraries or more precisely, actually, libraries, because we're targeting desktop as well. I get a similar talk by the same name last year at rusty days, but I modified the content, so there's gonna be new stuff. Even if you listen to the last talk.

A

So first a little bit about me, I'm a firefox telemetry engineer at mozilla and I'm gonna talk more about what that actually means, and I'm also a russ community. Member and beyond that I do organize the russ berlin meetup and also russ fest, the european rust conference and when I'm not at the computer, I'm actually trying uh to do more. Scuba, diving you'll find me online on twitter or my blog, and you will also find the slides under that link. It will contain all the all the additional links and material that I'm using.

A

So I'm here today to talk about rust and how we use rust at mozilla and, more precisely, I'm going to talk about gleam the library I've been working on for the more than two years now glean is a telemetry library. That's now used in our products as the data collection library, to give us data about how our products are used, which features work and how the performance is.

A

So why do we do? This? Are building a web browser and building a web browser means we need to make sure that the whole web works with it and it's impossible to test the whole web by ourselves. So we to a certain extent, we rely on getting some data about how the browser is used out there in the wild by all our users actually giving us the data with their consent.

A

So what what is this data? We try to instrument the browser to get different data points about the feature usage and how they perform out in the wild here on the screen. You see one of these data points which we call metrics and that's the metric which measures the time spent running the javascript garbage collector.

A

So, if you're using the browser throughout the day, whenever you visit a website that runs javascript at some point, the garbage collector will kick in and clean up memory, and then we measure how long that takes and then locally we aggregate that data and once or twice a day that data is sent to us.

A

Our servers will then take all the data we get from all our users aggregate that again and store that in a way we can later analyze- and here is one of the things we generate out of that data- a graph that shows the distribution of the time spent running this garbage collector.

A

This data is actually publicly visible on telemetrymozilla.org and what you can see here is a pretty standard distribution of runtime for the garbage collector and for the average person the garbage collector probably takes around 60 milliseconds.

A

So whenever we, our engineers touch the javascript engine, they don't have to only rely on their own local measurements on their own fixed hardware. But they can look at the data we get from all of you out there that are using firefox and see whether the changes they made have any impact for better or for worse. We can then use this data to optimize and improve our browser. The way it works, the best for users.

A

And when we do collect data, we do this under what we call the lean data practices. The lean data practices are essentially three principles. The first one is staying lean whenever we do want to collect data. We are very conscious about the questions we want to answer using this data and we state the questions up front and most of the time we actually upfront limit for how long we want to collect this data by just not having the data.

A

We might not be able to answer questions, but we also don't need to worry about how to handle the data. After all, the second principle is building security, as I said not having.

A

The data is obviously the best part, but we still want some of the data to make decisions, so we are very, very conscious about not collecting user data, but focusing on the technical data of how features are used and how the browser performs and again this is written out before we even measure any of this data, and it's also because it's all open source, you can actually check that out and the third principle is engaging our users.

A

We want users to understand why we collect this data and how this data collection benefits the product and how they can use the product after all, and also people are always free to enable or disable this data collection.

A

And last but not least, we do make some of this data public. We are very careful about how we do this. In general, you get aggregated data, as you saw before.

A

If you really want to know more about how we do this data collection, especially at the scale that firefox works at check out a talk by my colleague chutten about collecting data responsible and at scale. So now, let's get into the library that I'm working on glean glee. The clean sdk is a modern approach for telemetry library and we have like, what's called the bigger glean project that incorporates um everything from the sdk used in client applications to the pipeline and analysis analysis tools that we use.

A

About a bit more than two years ago inside mozilla the development of a new version of the firefox or android browser started back then we did have telemetry and data collection inside the desktop browser and, to a certain extent had that in the old version for android as well.

A

But we knew about issues was using that we knew that it was not the most convenient for our engineers to add new data collection or data points, and it was certainly not made for a modern mobile application. So the new development of firefox or android. We took as a fresh start to also redesign how we collect telemetry data.

A

So we were presented with a blank slate and could fix all the issues. We know that we don't want to do again. So what did we know back? Then? We definitely knew who's going to use our library.

A

First of all. Obviously it was firefox for android. That was the main target, especially more than two years ago, because that's what we were focusing on, but we also had a team on standby that was developing the ios browser that also wanted to incorporate better telemetry into their product, but the existing, inflammatory library was more thrown together than coherently designed.

A

So we knew we needed to support them too, and obviously, because we already have such a huge data collection library inside the desktop browser where we learned all these things that we don't want to repeat, we knew eventually we wanted to replace this.

A

So by now we already have three platforms that we need to target, but what's actually used there. How are they programmed well on android? It's mostly kotlin. That's what you by now use to write any application application, so you need to support an api in kotlin on iis. You obviously have to use swift to do anything useful really and on desktop, it's actually a mix of c plus javascript and rust. So these are all three languages that we also need to support.

A

So to us, it was very clear. Cross-Platform support is a must. We were a very small team of back then only four people that had to work on this and had to maintain it long term so re-implementing, the same design of a library in multiple different languages was just not feasible.

A

So we decided to build this on top of rust. So, let's jump into how the sdk actually looks. The green sdk is split into several layers, each doing their part to build the full sdk on the very low uh bottom layer. We have the gleam cork rate written in rust, which builds the all the functionality that glean actually provides.

A

On top of that we have the ffi library, which is the binding between the low level rust code and the higher level kotlin or swift code that is used later in the applications.

A

So, let's look at the glean core crate. First, the gleam core crate is really just a simple plain old roscrate. It has like a struct that holds most of the state. It has implementations on this struct that contains the main logic, including uh storing and reading data from the database, preparing things for upload tracking the metrics and especially also tracking how it behaves internally and then offering the api to the outside. To use this whole thing.

A

Additionally, we have implementation of all the metric types that are used um also written in rust. Metric types can be as simple as a counter as a string, but can be even more complex, like the timing distributions, as you saw for the javascript garbage collector before so that's all handled within a single rest grade, and there the benefit of using rust, already pays off because we're all written it's all written in rust. We can use everything that's available in the rust ecosystem. We can rely on external crates.

A

We can rely on the tooling that is available for us, and testing is also super easy and super quick and additionally, we we already support all the major platforms right from the get go which we can compile to, but obviously that's only half the story, so let's jump upwards in the stack and look at the clean, ffi crate.

A

The ffi crate is the connection between kotlin, swift or other implementations to the lower level rust grid. Now, most other languages can't really talk directly to roscode, but most other languages can talk to c code. So by providing this ffi layer, we translate rust into c, which can later get called. So what does f y stand for? Ffi stands for foreign function interface and it's really more or less just a deterministic and simple naming of of symbols such as the functions that you can call, and additionally, it is a stable abi.

A

That's the application, binary interface, which describes the memory layout of the data and how you call functions and pass data into these functions and get them out.

A

Writing a ffi layer or c compatible layer in rust is actually surprisingly easy to be callable from other languages. That no rest. That no c you have to write a bit of plain rust code and extend that with a couple of attributes to make it possible.

A

Here you see a function called header, rust lens, that is actual rust code, and that has several additional attributes. First off you see nomangle nomangle signals to the compiler to take the function name as is and put it in the binary, so any caller can find it by that name later.

A

If you don't do this, the compiler will actually change the name and encode a bit more information into the name, but make it unpredictable what it is and therefore you can't call it. The next attribute you see is the extern c.

A

This specifies the api that is used when this function is called so how data gets passed in, whether that's on the stack or in registers, and how data gets passed back out and who's responsible for managing certain parts of the stack and registers.

A

Now the cabi or c itself is much more restricted in what it can express and is especially more restricted than rust. It doesn't have the expressive types that we have in rust. It doesn't have generics and it doesn't know references and this. This shows you will always need a translation between actual rust types and the types as they would look and see to pass data around, and one thing you already see on this slide here is strings. Look vastly different in sealant than they do in rustland.

A

Strings in sealand are really just a pointer to a bite buffer somewhere. That's ended with a nullbyte, that's completely different from how it looks in rust.

A

Luckily, rust makes it easy to work with strings that are coming from c or need to be returned to c there's c string in the standard library c string. Just takes your regular rust string, appends a null byte and then gives you access to the underlying buffer with a or even a pointer to this buffer. So you can pass it back to c.

A

There are more of these things in the standard library available that help with some of these abstractions needed to talk to c or get called from c now. One thing if you want to get called from c is that c needs to know what functions are available and how to call them. So, usually you have the definitions of your functions in a header file that you include so at compile time the compiler knows what to expect and the linker will later link to your library, which has the actual implementation, but writing out.

A

This whole header file is a bit tedious, especially if you're writing a full library. You don't want to do this for every of your functions.

A

Luckily, for us, there's a tool out there c bind chin c by chain, takes a rust code, parses it and then generates the appropriate c header file, so consider the code on the left. Our hello, rust lins function again again marked with no mangle in extern c and returning a pointer to some arbitrary data.

A

If you pass that to c9 gen, you get a header file that looks like the one on the right. You got a c definition of the same data of the same function with the appropriate c types. Now, if we take this and throw this against our library, we get a large file that contains everything we need to make this callable from anything. That knows how to call c.

A

Luckily, on top of that, we make extensive use of the abstraction functionalities that rust has to offer, and one of these things we use is ffi support, a library that is that essentially bundles a lot of the patterns we used throughout our code base.

A

It was started by another team at mozilla, the application services team that do very similar thing of writing, cross-platform libraries and they come up with some nice patterns that make it easy to write ffi code. That looks mostly like rust, but under the hood, does all the necessary stuff to be callable from c one of these nice abstractions we use is the inter-ffi trade intel, ffi generalizes about type conversions between rus types and ffi compatible types.

A

I already talked about strings and how different they are and how there are certain obstructions in the rust standard library, but passing strings back to c is still a mostly manual process by using into ffi. We can write this process once and define the behavior we actually want, and then in our code we only need to use this trait and call into the straight instead of repeating using c string manually. All the time.

A

Another nice abstraction that we use is ffi screen. So far, we've been passing strings back, but we can also take strings from the c side into the rust side. But I've said before strings are really just pointers and pointers in rust. Don't have any lifetime attached. So you need to be very careful how you handle them. You can't really store them and you definitely can't turn them into reference to then store them, because the data behind this might be gone by the time your function ends.

A

So you will always need to clone the data and then turn that into a rust type and make sure to use it that way, but nothing in the pointer really helps you with that. Ffi string changes that ffi string essentially is a wrapper around such a string pointer and make sure that it will have a lifetime.

A

When you take this as a parameter on your function and also restricts how it can later be used, it will only allow you to get a reference for the short time that your function runs or get an actual own string that you can store somewhere else by copying. The data out.

A

The nice thing about these two abstractions that I shown you right now is that they are really just zero cost abstractions. They compile down to the exact same code as if we've written this all manually, but we we can rely on the compiler to do the right thing if we've written the right code once so. Let's look at compile targets.

A

Next, most people write, compile and run the ras code on their own machine. Sometimes obviously they need to run it elsewhere and that's when you need to reach for the compiler to actually compile for that other target. Think about compiling on your mac and running on a linux machine.

A

Luckily, the rus compiler is a cross compiler by default. The rest compiler knows how to take ros code and turn that into some other machine's machine code. That only applies to roscode, though, and there is a bit more to actually get final working library or binaries.

A

So if you're installed rust through rust up, then you can ask rust up for a list of all these targets. That rust knows- and it will show you a long list and will tell you which ones it has installed and by installed here it really just means it downloaded a compiled artifact for the standard library, so you don't have to recompile that every time, so you might look at this and think well, what is what is this?

A

What are these characters? What does it even mean for something to be arg or x86 and then unknown? So what is the target?

A

Well, the target is roughly the combination of the computer architecture, the operating system and the api you want, though, sometimes the abi is actually the lip c you link against, and sometimes the target is a triple, but has like three to five components depending on who you ask, and yes, I'm not totally ignoring any of one or more errors here and as I've recently learned, some components of this triple can actually contain version numbers which, down the line in your tools, decides decide which other tools to call it's quite a mess.

A

Sometimes the glean sdk is currently compiled for 11 targets. We have several targets for android here. We then have targets for all the major desktop operating systems, either for testing or even for shipping our python bindings, and then we have the ios targets.

A

Adding them using rust up gets you mostly there to compile your ras code.

A

It can compile the rust code, but that doesn't really mean you get a successful build out of that rust doesn't know anything about the target, specific linkers or libraries. So most likely, you will need few additional tools to get final working libraries for android. That's the android, sdk and ndk easy to get by for ios, that's a mac and xcode which will bring the sdk and tools that you need.

A

So now, let's wrap up the ffi layer and look one further up. Let's look at our glean kotlin implementation we're only focusing on this year, but the other implementations actually mostly similar.

A

So how do you make kotlin talk with the c api? We use a library, a java library called jna.

A

Jna provides java programs easy access to native shared libraries without writing anything, but java code, no jni, which is another library or native code, is required when you read java here think about the jvm, it works the same with kotlin, which is what we use.

A

Let's quickly jump into a small hello world and how this looks with jna. Again we have a rust function with our additional annotations, no mangle and extern c, and we return some pointer to some string data. That's our ros code! On the kotlin side. We have we use jna and describe our interface. Then we tell jna how to load this using native loads and giving it the name of the shared library further down. We list all the definitions of the file of the functions that are available.

A

I spoke about c binding before which generates c headers. Unfortunately, kotlin and jna are not able to read these so here we need to manually list out the ffi functions with their kotlin equivalence or jna equivalents manually.

A

It's a bit bothersome to do this, but it's only for one of our bindings, so we did it once and it's it's not perfect, but we just didn't invest time to build tooling around that.

A

If you have done this, you get an instance of uh of us of a class in kotlin or java, on which you can just call these methods as if they were native kotlin.

B

A

And that's really just it to compile this you're going to rely on gradle the build system that is used throughout the kotlin and android ecosystem.

A

You can use the rust android gradle plugin, which takes care of compiling stuff for you.

A

It gives you a configuration block where you configure, which cargo project to build where it is by what name it you, what what name the library will have when it's compiled and the targets you should compile for now whenever you build your android project, gradle will also build your cargo project and make sure to place the final dynamic libraries in a folder. So your test code or your application will.

B

A

A

And that's basically, all there is to it. We do a lot more stuff on the kotlin side to then build the nice kotlin api, but internally we're just calling any of these functions on the api on the ffi layer, and we do this for several other clean implementations as well.

A

As said before, we do have a swift implementation and there it's a slightly easier because swift speaks c, so we can use the c header we generated using c by gen more easily.

A

We also have a type implementation which works similar to swift, using cffi again and parse, the c header, which internally generates python code that gets called, and then we have a bit of python code that provides the nicer api to users of the sdk.

A

Next, we have a c plus implementation which is used in firefox desktop. It works slightly different because of the setup of our desktop works and how it interacts with clean, but conceptually it's very similar. It calls some ffi that calls into the clean core crate everything is compiled together, so everything just works, and lately you also have a javascript implementation for desktop now. This is actually also written in c plus it's one of the benefits of being the browser that has the javascript engine. You can write your javascript code in c plus, but don't worry.

A

This is not directly accessible to any websites. It's only for the javascript that actually builds the browser itself.

A

We do now have another project, essentially a re-implementation of the glean sdk called gleanjs, though glenjs targets, websites, web extensions and even node environments, so you can use that there as well. If you can't rely on the glee native library and last but not least, we do have a rust implementation now. What I didn't really show is that the gleam core implementation is very low level. It's just enough to provide everything to the ffi layer, but doesn't provide the nice api that we want our users to see.

A

So we have another implementation of glean on top of the glean core crate, also written in rust.

A

That has a much nicer api to use, but you see from this list there's a lot of things we need to target and whenever we change something in the clean core grade, we also have to re-implement that up the stack whenever we add a new metric type inside the core grade, we need to add functions to the ffi layer, to use this metric type and then build a nice api in kotlin in swift and python and the other languages just to make it usable across all the implementations and that's still a lot of code.

A

It's not a lot of logic, so it's more like copy and paste that we do, but that's still not nice to maintain.

A

So what? If we don't need to write that much code? I always think about that, and luckily others at mozilla also think about this and actually find the time to do something about this. So some other good folks at mozilla came up with uni ffi uniffi is a multi-language bindings generator that essentially everything that I just told you does under the hood, and the only thing you write is rust.

A

So let's look at this quite quickly before we end using uniffi, you define what your interface should look like in a web idl, like definition, language web idl, is used to describe, for example, the javascript interfaces as they are exposed to websites and it's extensively used in both firefox and also chrome, and we can now use this to describe the interface we want from our cross-platform libraries.

A

So here we define an interface for a rocket. A rocket can be constructed with a name it can then later be launched and because the launch can obviously fail, we can throw a launch error.

A

We now take this definition and implement it in rust. This is again pure us implementation. We have a struct that holds some state about the rocket and then an implementation of the rocket by convention. The constructor becomes the function, mu and other functions on the interface, become normal methods on this struct and throwing in our interface turns into a result type on the rust api.

A

Other languages that use this later might see this as an exception or the equivalent to a result, type now equipped with thrust implementation. In our definition, we throw this against unifi and some magic happens, and when that magic magic finishes, you can use this in swift in kotlin and in python just as easily, and we didn't write a single line of kotlin, swift or python to make this all work.

A

Unifi did that for us all, we did is use a nice api that feels very ergonomic in the language, we're writing it in and under the hood. It does all the right things to call into rus code and actually run so what's this uni f of I magic?

A

The very first thing, obviously, is the ffi layer for in gleam. Currently, that's a whole crate. That's doing this in uniffi. This is just generated for you and does the right thing all the time the next thing unifi takes care of. Is the conversion between all the types again you need to convert the rest types into something you can pass over a c.

B

A

And then later need to turn that into something the target lagrange understands, and it uniffi knows that in both directions.

A

So whether you return a string from rust and use it in kotlin or generate a string in kotlin and pass it into rust, uniffi will take care of allocating the right amount of memory to put data in pass it around and later de-allocate it all safely, and then it also provides the convenient apis for the target language. So it generates code that actually feels very ergonomic in the language you target like python, kotlin or swift.

A

So how does this look like under the hood? Well, this looks actually more or less the same as what I just described to you in all the other slides. You again have some functions which have our now known attributes, no mangle and extern c.

A

And if you look into the actual last line of code, you see the call to rocket new, which is just a call to the actual implementation of our rocket that we wrote ourselves and then it does a bunch of other stuff and has some weird naming on the other side of the ffi.

A

You have the swift code, where it just implements a swift class and the constructor of that swift class knows which ffi function to call and how to pass data in. It also adds all the other functions and also knows how to call the right fi functions for that and how to pass data in there.

A

And that leaves you with writing. None of this at all. Now, we've wrote code exactly like that before, and I give it to you like. We used a bit more human, readable names where uni ffi mangles these names again, but they unified.

B

A

Track of it, so why let us write that code? Computers are much better at it. So last thing you might ask is so: why did clean do all this stuff? Well, uniffi only exists because we did all that stuff manually first and now that we have more than two teams doing this. It was clear we can't continue writing all this manually and we needed to do this for us.

A

So when do we get this for gleam?

A

Well, whenever I find the time, but hopefully by the summer, uh we will start moving over glean to uniffi, instead of maintaining all that code thanks a lot to the glean team at mozilla, my colleagues lsu br chris travis and mike who helped maintain this project with me and also to the application services and uni ffi team who we are in constant exchange with, because we're still tackling the same problems together on the slides you find links to all the to some of the resources glean and all the work we do is all public and open source, so you find it there and now, if you have questions now, would be the time to ask.

A

B

Thank you very much for this super interesting talk. For now I will move the slides away and uh yeah. It was very interesting and first let me say thank you for taking the time and presenting to us. um I see people are currently typing in discord, so maybe I can start with one that came up to my mind when I was um listening to your talk, you talked about how kotlin and swift can call into rust to do something.

B

There do you have use cases or the necessity to somehow call from rust into, for instance, the android operating system into java or uh or things like that? Is that also a use case, or is that something that is not relevant for your project.

A

So in my project directly, it is not necessary um because the way our telemetry is set up, it's pretty much a one-way thing where you pass in the data that you record and never need to ask for it back because everything else is handled internally.

A

So we don't need to support that which takes a lot of work away from me, because I don't have to think about this, but um obviously like other parts um and the other library to a certain extent. They need to do this. um So there are ways to do that. For java, there are ways to call back into um or for the jvm there are ways to call back into that part. It's um not a streamline as the other way around currently, but it is not impossible for swift.

A

I don't actually fully know, but you can also like do essentially the same as you would do with rust, where you have like the ffi, like abstraction, on the swift side, which gets callable from the other side.

A

And then we have some work that actually allows to do some form of callback mechanism, so it can actually register callbacks that get called in one or the other direction, though that has a few more problems because of how memory allocation works and whether you need to be very careful with the garbage, collector and stuff like that.

B

Okay, thank you. Thank you for this answer. Now we have a question. No, this is the wrong one. This one we have a question from bram. It was coming in through the discord server. How does unified deal with cleaning up resources from garbage collected languages? He was asking whether it does it the same way that wasn't buying gen. Does it so the calling language has to explicitly drop or anything you can. You can elaborate on that.

A

Yes, so that is, that is actually the nice thing that uni ffi takes away from us so right now, the way this is, for example, handed in kotlin and in swift is that you need to explicitly use some constructs to allocate data on that site. um That's mostly unsafe.

A

Actually, um they have similar concepts that as rust, so you allocate data there and then pass in the buffer into the rest site or through the ffi into the rust site and rust can fill that data and then the kotlin or swift are later responsible for cleaning up their data in swift that has like unsafe apis that do this for you, which basically um work around their garbage collectors. So you need to be explicit about the allocation and the allocation there on the java side, it works essentially the same.

A

It also has makes this very explicit in the current glean code. We need to do that all manually and we have these allocation calls. So our coupling code actually looks very c-like because it needs to do that stuff.

A

um We try to hide that behind some abstractions, especially like for strings where we do this, but that's essentially what unified does behind the seats. So whenever it calls and needs to pass in data, then it will allocate put the data in pass that over and later make sure that before it returns to you, the data is deallocated.

B

Okay awesome. Thank you very much. Thank you very much. I have uh one question from my side when, looking at your talk and seeing how you use the web idl to specify interfaces, I was thinking about how I currently work with rust and wasn't buying jen and wasn't pack and so on and and became aware that it is kind of a little bit different, but also a little bit similar.

B

Do you see any relationships between these two projects, so the c bind gen and the wasn't buying jan and the wasn't pack and the web idl use in your project.

A

Yes, um uniffi was very much inspired by the work that wasn't by engine. Does uniffi also takes a lot of inspiration from some of the interface type proposals for webassembly itself, which specifies how you encode certain data, so you can pass it over a much more restricted memory buffer part.

A

The reason we do or unifi wants you to use the idl is that this allows you to specify your interface up front and we can generate the same interface on both sides.

A

So the interface that you describe is also what you need to implement in rust, which makes the rust thing directly usable as a rust crate without the needs. Another crate on top, as we currently do in glean, but it also allows you to create the c plus bus codes and the python code and the kotlin code that will look like actual python code, which just internally, does all that stuff.

A

So there's definitely some inspiration there and yeah. The using of the interface definition is is definitely um one of the big separations between these two things.

B

Very interesting, interesting thoughts and um it's interesting to hear how these things, how these projects inspired each other young eric. Let me share some things at the end. You see we have really great um feedback coming in here we have one, and here we have one and many other things are going on at discord. Thank you very much for being here.

B

um If you want, you, you're always invited to stay a little bit longer on the discord, server, ask or answer questions there or whatever, and whenever you have something interesting, you would like to share with us. You're always welcome at as a speaker at the ross plains meetup. So thank you, young eric for taking the time, and thank you thank you for coming at least virtually from berlin to lynch today.

A

Thank you for having me it was fun and thanks everyone for.

A

A