Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tower of Babel: Making Apache Spark, Kubeflow, and Kubernetes Play Nice - Holden Karau, Netflix

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tower of Babel: Making Apache Spark, Kubeflow, and Kubernetes Play Nice - Holden Karau, Netflix

Working with big data matrices is challenging, Kubernetes allows users to elastically scale, but can only have a pod as large as a node, which may not be large enough to fit the matrix in memory. While Kubernetes allows for other paradigms on top of it which allows pods to coordinate on individual jobs, setting them up and making them play nice with ML platforms is not straightforward. Using Apache Spark and Apache Mahout we can work with matrices of any dimension and distribute them across an unbounded number of pods/nodes, and we can use Kubeflow to make our work quickly and easily reproducible. In this talk, we’ll discuss how we used Apache Spark and Mahout to denoise DICOM images of lungs of COVID patients and published our Pipeline with Kubeflow to make the process easily repeatable which could help doctors in more resource limited hospitals, as well as other researchers seeking to automate the detection of COVID.

A

Oh cool, I guess I'm giving a talk: okay, rocking ooh!

A

I can hear myself fancy cool, so I am here to talk about the tower of babel or babel, I'm american, so I can't pronounce words actually I'm canadian living in america, so I extra can't pronounce words, um and this is about making apache spark apache, mahout, kubeflow, kubernetes and a few extra friends all playing nice together um and now. Every time I hear a tower of babel or babel, I think of the book snow crash. Has anyone here read the books, no crash?

A

Okay? So the three of you are going to love the references that I'm going to make for the rest of you. Just pretend that I am hilarious um and that pizza is somehow related to machine learning.

A

Also, you should consider reading snow crash, it's an excellent book with some weird bits in it um and it makes me very hungry for pizza, so here's a picture of some pizza, so you can be hungry too. Okay. So, in addition to not being very good at pronouncing words, my name is holden, my pronouns. Are she or her it's tattooed on my wrist, which is really convenient in the mornings when I'm just like what? Where am I, who am I um and I'm on the apache spark pmc?

A

What that means is I'm like a committer, but I'm really really hard to get rid of. Unfortunately, it's not like tenure in the same way uh as it guarantees that I get money I can still get fired and in fact I checked my email right before this talk to make sure I wasn't part of that two percent um good good news um and I contribute to a lot of other projects. Besides just spark um previously, I've worked at a whole bunch of other places.

A

I haven't yet quite sort of caught all of the pokemon um in the traditional bingo card that you get in in san francisco, but I'm confident that before I eventually get hit by a car and die, I will succeed at this. um My co-author of a few books, including one that is actually related to this talk, and you should definitely buy several copies of each book. They make an excellent gift for whatever the next holiday is, and this is europe's you have tons of holidays, so you should definitely buy tons of these books.

A

You should also follow me on twitter, and, if you like, questionable code, you should check out my github, as you can tell by the jokes that I'm making, I may not represent the views of my employer, although they definitely do pay other people to make very interesting, jokes and I'm realizing. I should stop talking. um We are hiring still um and if you're interested please reach out, although it's mostly in north america, so if you like, paying for health insurance with a credit card come and talk to me, no okay worth a shot. Okay.

A

So, in addition to who I am professionally, I'm trans, queer, canadian in america on a green card um and part of the broader leather community. Now this is not particularly related. There is no secret canadian um out of memory. Exception, debugger ring that they give us right. um It's just.

A

I think it's useful for us to all talk about where we're from so that if we realize that we're all surrounded by folks with very similar backgrounds, we try and get some other people in the room, and that's also actually part of why I come to europe. It's not just because I enjoy beaches.

A

It's because, like I like meeting people from different backgrounds than myself, my co-author is is not present. Trevor is a wonderful person um he is based in chicago. He has a new kid. It's very exciting. Everyone send like happy, warm vibes and maybe getting to sleep sometime this year, energy towards trevor.

A

He is the pmc chair of mahut, which means it is even more difficult to get rid of him once again does not come with any guarantees of money, just really difficult to get rid of, and he's an asf member right now he's mostly looking after his kid, but he is also trying to import electric tricycles into america and if anyone happens to be interested in that, which is a pretty long shot, definitely reach out to trevor.

A

His email is at the end of the talk, um and you know that's probably a great reason to go and visit chicago, which is not as nice as here, but does have water, okay cool. So what are we going to talk about? So we're going to start with our adventure?

A

um Slash case study, but I thought adventure sounded cooler, so act. One is going to be getting to know the characters and the problem that we're trying to solve. Then we're going to talk about the problem in like a little bit more depth and then we're going to talk about how we solve the problem, um and then I mean I should use air quotes when I say solve solved, except by the time we solved it.

A

The solution wasn't useful anymore, um but that's okay, because we did a bunch of cool things along the way, and so that's going to be the epilogue and we'll we'll talk about the cool things that we can learn from our exciting adventure.

A

Okay, who are our friends um besides on twitter, so okay kubernetes, my second favorite friend, uh cube flo, who is the main character and for the three of you in the room who read um snow crash hero protagonist um for the rest of you. That was a hilarious joke. I don't know why those three people aren't laughing, but maybe they're european um and apache spark who is my favorite, but in in kind of the way that your kid is your favorite, even if they're like, maybe not so good at everything.

A

You still like, really hope that they're going to succeed this time, that's sort of how I feel about spark and apache mahut, which is not my kid and therefore I care a little bit less about sorry. Trevor um apache mahoud is very much trevor's kid and and much more important. um Okay. So I'm going to assume that this is kubecon, we're we're several hours into it.

A

You probably know what kubernetes is um so we'll sort of skip past that if you're new to the cube community, that's awesome, I'm super stoked you're here um and that's really cool, but but we're not gonna. We're not gonna, explain kubernetes um in the context of what we need it for in this sort of machine learning thing um how many people have had to work.

A

Sorry have had the opportunity to work with data scientists, okay, cool, how many of you have gotten something like untitled underscore x, dot, ipython notebook from data scientists that is almost the same number of hands? Okay, and how many of you had that run successfully without needing to install any dependencies?

A

That is no one great okay, and so that is why we are using kubernetes. um Also because running on one computer is slow and also kubernetes is cool and we like money, uh so cube flow is if someone was like yo what about? If we like, put all this machine learning stuff on kubernetes. So it's got this cube and flow.

A

We're going to make pipelines it's going to be really cool. Everything is definitely going to work and you should definitely buy my book about how it works and we need it because putting together all of these different tools, kind of sucks right, like no one- really wants to be like sitting around waiting for a job to finish, to go and kick off another job to go and kick off another job right. No one really wants to like have to think about how they're going to get their data from one tool into another.

A

We just want some magical tool to take care of it and kubeflow promises to do that, for us we'll find out later that we did have to trade much of our happiness. For that, but that's okay. I trade my happiness for money quite frequently.

A

The other reason why we need it is because reproducible, research and grad students hate reproducible research almost as much as grumpy cat and kubeflow. Is this wonderful opportunity to make it so that we, incidentally, get reproducible research out of things? If we build them in kubeflow, we can get these nice pipelines and we can run them again in the future. Once the grad student graduates, once our co-worker wins the lottery once their share's finished vesting.

A

um There are some other jokes here, but we'll we'll move on we'll move on yeah, yeah, yeah, okay, cool, so spark how many people are not familiar with apache spark.

A

Oh four people you should buy my books, but for you, four people, I'm so stoked that you're here, yeah okay, so spark is a really cool data processing tool and it definitely works 100 of the time it works 80 of the time, um and so it allows us to do distributed data processing. So we can handle data, sets that are too big to fit in an excel spreadsheet.

A

um So if you find yourself trying to open something in excel and excel is like hey, I can't open files from hdfs that is apache spark we're gonna, we're gonna, make it better.

A

Okay, yeah and we need it because it turns out that um ct scan images are kind of big and don't fit very nicely in an excel spreadsheet and, to be honest, they don't fit really nicely in a computer um and spark is able to handle everything from doesn't fit in an excel spreadsheet. All the way to doesn't fit in several computers on the flip side spark does a really bad job of handling fits on a floppy disk.

A

um So if you have data that fits on a floppy, disk spark is probably not for you, but you should still buy my book. Okay, apache mahout, oh wait! We've got the yes new logo.

A

Don't worry, none of the code has been improved, um but the new branding is glorious, glorious glorious. No, and I joke I joke. Actually much of the code has been improved. Apache mahou was originally created for mapreduce, you know, kicking it old school and then spark came along and people were like whoa mapreduce isn't cool anymore. Let's use you know, spark very happy about that decision. Y'all um and so mahu was like. Oh okay.

A

We should rewrite to spark that took several years, because people are lazy and I'm including myself in this I'm lazy too, um and then they got a new logo so that you could know that it was new and fancy and ran with the new fancy thing that was about five years old. Okay.

A

So the other thing about apache mahout is apache. Mahout is a tiny, tiny little project that refuses to die largely because of trevor trevor is amazing and if you have ever said to yourself, you know what I want to get involved in an open source project, but there's just way too many people in all of these projects. It's going to be so confusing. I will not know who to talk to you should get involved in apache mahou cause you can just email trevor. He is always the guy to talk to 100 of the time.

A

I'm joking. There is actually a mailing list, but you can also just email trevor. It's pretty fast, sorry trevor, okay, cool, so why do we need it?

A

So we need it because math is hard and the people that wrote spark, including myself, are kind of lazy, and we made some machine learning tools but then part of the way along the way we found out that, like people were giving us money, anyways and so like, maybe someone else could make the machine learning tools, so we kind of stopped making them, uh and so that's why we need mahout, because we want to do some fancy machine learning, type things and some fancy math on top of spark.

A

Okay and s3 buckets. How many people here are new to s3 buckets, I'm so sorry.

A

Slash, congratulations.

A

These are bad, but it's okay. They beat the alternative of doing it ourselves, so we can store data in them and sometimes we can even read the same data that we stored from it. Not a guarantee certain terms and conditions do apply, not valid in us east one. um Yes, okay, at least someone likes my cloud jokes, um so yeah they're, usually not the most performant, but the alternative is standing up my own hdfs cluster and if you've ever stood up your own hdfs cluster, you too will be very happy to use amazon s3 okay.

A

So what is the problem that we're gonna solve we're gonna solve the problem, which is why we're all wearing masks um to be clear. We don't solve it. We all still have to wear our masks except for me because I'm talking, um but so the big problem in the early days of covet was we didn't really have a fast way to detect uh if someone had covet, and so we wanted to do coveted screening- and we thought you know what I have a problem: let's add computers, and then we had two problems.

A

So, let's see if we can solve the problems we created for ourselves, the answer is sort of okay, so we needed rapid testing. um So we're going to go back to march 2020 when you could not like walk into a walgreens. Oh crap, you don't have walgreens here uh boots the thing with the green plus sign pharmacia. You could not walk into the pharmacia and buy a covid test. um So back in march 2020 life was sad. um Life was very sad here.

A

Yes, life was very sad in america and we couldn't really figure out like who had coveted and it took way too long. So, um oh yeah and the new rapid tests that we got were about 60 accurate, oh yeah, slightly better than my average. In my non-major subjects, um shout out to google for not checking my gpa um or netflix too okay, yeah and so really cool. A bunch of people came up with things that were more accurate than 60 and went kind of you know. Kind of fast um ultrasounds were pretty cool.

A

The ct scans uh showed a lot of promise now, admittedly, the people who said that the ct scan showed a lot of promise were the people who did ct scans so like possibly some bias in the same way that I might tell you, apache spark is really cool and you should buy my books um so yeah best diagnostic tool, according to the person selling you the diagnostic tool. So that's great. um There were some slight slight problems with that.

A

um One of them is cancer, and so it turns out that there's some downsides to getting a bunch of ct scans, uh and this is radiation and so to like detect uh covert in someone.

A

Initially, you needed to do a full body like fairly high dose ct scan right, and that's that's not great, especially if you might be doing it multiple times on people right. I don't know how many people here have taken more than one covid test. um Certainly I have a lot more yeah right, so that was definitely like. Well, okay, we should we should see if we can like make something that isn't going to turn the population into like little radioactive people.

A

um So, instead we could use low-dose ct scans right and the the plan was more or less. What we're gonna do is we're gonna go like csi miami, which I really hope you have and we're gonna say enhance and we're gonna turn this into something that tells us what's going on, um and so in comparison, much less radiation, much less chance of cancer yay only problem.

A

We need in science fiction technology. Okay, um the good thing is, it turns out that some people who were way less lazy than us um came up with some ideas to denoise images a long time ago. um It turns out, though, that, like it's kind of hard and denoising them revolves about 500 gigabytes of ram um if we wanted to denoise full, like body ct scans, and it turns out that my credit card has what would be described as a low limit.

A

So, like that's not happening, um and we should like figure out some way to do this without using 500 gigabytes of ram every time we want to dino as an image.

A

So we figured okay, you know what I've got a problem, we'll apply machine learning to it. It'll give us magic technology from the future everything's great and we'll run it on kubernetes, so that'll be cool too. Okay, intermission.

A

Okay, no one likes my intermission music. Okay, so we we yeah, wait! Oh no, okay! There we go yes, so we we really needed some way to do. Detection of if people had coveted the best idea that we had was, admittedly from someone selling ct scans, but it seemed like a really good idea um and in fact there were a whole bunch of people who did uh collect data on sort of what the scans of people with covet looked like now.

A

To be clear, we were not like hey we'll make a model, that's going to like tell you. If you have coveted or not, there are a bunch of people who tried to do that and it turns out they did a really good job of detecting. If someone was lying down when they were having a ct scan, because they were much more likely to be lying down in a certain position if they were really sick um and so yeah correlation and causation not the same. But you know: okay, cool cool we've got data set.

A

We've we've narrowed our scope of problems, so we're just going to use machine learning to make the images. Look better and then we're going to give it to a human. So it's not our fault. If a bunch of people like uh don't do so well, one of my goals in life is to not be directly responsible for someone's death um right, okay cool.

A

So why did we use free and open source software for one thing, I'm cheap uh for another thing I I like using open source um and for another thing at the time things were not looking great for people who had kind of low limit credit cards, and especially for countries who had kind of low limit credit cards, um and the nice thing about free and open source software is like yeah, it's free.

A

Admittedly, it's only free if you value your time at about zero dollars, but conveniently that's what I value my time at, um as does trevor, unless my employer is listening, in which case please continue to pay me money. um My time is worth whatever you pay me, plus ten percent, okay cool.

A

So we had an idea we were going to make a pipeline. It was totally going to work. Everything was great um and because we were also, admittedly working on a book. At the same time we were like you know what would be really cool if we did this with cube flow, because kubeflow could totally solve this problem for us so we'd take our ct scans. We'd turn them into something that we could do.

A

Our like: fancy, science on um we'd load them into spark and then we'd do our distributed uh svd and then everything would be great um and then we could de-noise our image and then, in theory a human could look at it and say: like yeah, this person looks cool, they just have a cold or like e. This person should not like go outside right now and they get to go into the special room, um cool. So first step.

A

We loaded our data, and so the wonderful thing about being lazy is that you search on pi pi before you write code, um and so the images were all in a format that we couldn't just directly load into kubeflow great news. Someone had already written the library for it because it was in python. We could just really easily whip up a python script, make that the first step of our pipeline and it took the images and dumped them to a pvc now. This did have some slight implications for scaling in that.

A

No um because we had read, write once uh pvcs, so so we were, we were only able to run one instance at a time, but the good news is that the data set at this stage had not yet become the like holding this very sad stage. So it was okay that this was not happening in parallel and it was only happening on one node.

A

The next stage is the one where, if we weren't doing it in parallel, life would be sad, so we read them in from disk into an rdd. It is kind of annoying to do this in spark something that we should be doing better because it's reading from a local disk into a distributed disk. One of the things that I really wish was easier was, if we could have uh read, write once into read many conversions for pvcs in a happier way, but that's a long story and not particularly related okay.

A

So um and then the drm thing is not the thing where they didn't want you to listen to music in the 90s uh or early 2000s. um It's some mahoud thing. I think the m stands for mahout might stand for matrices. I don't know this was the like sciency part. So trevor did the math part okay, and then it came time to do our fancy math.

A

Okay, so svd is uh yeah, that's hard and um let's see how many minutes I have left yeah, not that many, um because I don't have four months so we're not going to go into the details of how we perform an svd. But suffice it to say mahout on spark can do happy svd and it's it's very nice. It does not need 500 gigabytes of ram on one computer.

A

Everything is great, no rusty, spoons required and if you're interested in actually learning what's going on here, there is a link at the bottom, and these slides are actually also on the schedule link. So you can go to the schedule link and then you can click on these links, so you don't have to write it down or take pictures of course. So, like feel free to take pictures, because boo is fabulous. Oh yeah, boo is my dog okay.

A

So one of the things that happens, though, sometimes when we do things is we're, like you know what we should do, we should see if it worked and that's like always a mistake right, that's the first step to sadness um and yeah, so yeah it was sad um or more specifically, one of the things that we realized is well like yeah. We could just run this iteratively like a whole lot.

A

uh We would eventually like go from like kind of okay image to like slightly better image, all the way back to kind of crappy image, um and so like humans. Humans need to do things and we can't just give the magic computer box the magic button and everything gets better um in part, because we don't have like a good enough. Fitness definition of you know what a good image is like that's kind of humans, okay cool, so um we did have this pipeline. It cleaned up a bunch of images.

A

What what happened?

A

um Or you know what came out of doing all of this stuff so before, like these results were published, the world changed and and for the better to be clear right like if we were all getting ct scans. You know before getting on an airplane that would kind of suck um tests got cheap, a lot cheaper than the cost of doing ct scans and they got more accurate than 60, um so pretty solid. So, just like a real software project in the time that we finished it was no longer useful.

A

But you know there were a bunch of really neat things that sort of came out along the way um one of them is like there was this idea in the early 2000s that we could do this, like futurey sciencey thing, and we could clean up these images.

A

um There was a published paper, not enough code to like actually really just like, go and run it, but we were actually able to recreate it and that's kind of cool right and like reproducible. Science is like yay, it's it's happy right. Grumpy cat and grad students might disagree, but that's okay, because grumpy cat is not the principal investigator um right. Okay, we also like, along the way, discovered that running spark on kubernetes is a lot of fun, yeah yeah, which is part of how I have a job.

A

um So one of the things that we discovered is that um we didn't have a shuffle service on co cube, and so we worked on some alternatives um and I changed jobs in between um and the first alternative that we came up with.

A

Was this kind of janky decommissioning thing where we copied files around um then one of my co-workers came up with another, even smarter thing where we would copy files around and then, if, like things, went to hell, we'd copy them into an object store and we could still scale to zero uh or not quite zero. We could scale to one because there was certain information that we couldn't manage to get out of the jvm. Oh yeah, we use java, I'm sorry um other exciting things that were admittedly unrelated but were pain.

A

Points that we experienced that have been improved is pod allocation for our scale. Down is kind of flaky um and scale up, so we're scaling down and back up and down and back up- and this happens a lot when we're doing things like switching from etl to the machine. Learning phase um really excited that volcano and unicorn support is now in spark.

A

I think it's being voted on right now, although it did just get a minus one uh during the previous talk, so like no, no, no like hard commitment on when it's going to be available. But if you like, building from source- oh yeah, you can, you can play with it and then video folks did some wonderful things with making it so that the training and etl stages can use different types of resources um instead of just like having a different number of machines.

A

Spark can actually take advantage of the fact that kubernetes can allocate pods on different machines with uh different kinds of resources.

A

um The other one is that we got an example for our book, which, once again you should all buy, and one of my co-authors is actually here in the in the road. If you want to yeah, so you should buy it so that we can each afford more coffee um and you should buy several several copies, um but yeah there's there's the example on github. You can check it out and it's a full pipeline. It's not just like an ipython notebook and a note that says: good luck have fun um right.

A

You should buy these books uh they're great for the holidays and make an excellent gift. There is also the the article that came out of it. I get no money from the article. Do not care if you read it um the book, though yeah that's where the money is okay, cool. So what what could we have done better right like if I had a time machine or if I was solving this problem again today? Let's hope we don't have another pandemic.

A

um What what could we do better, so cube flow makes using all of these different tools possible right and that's kind of cool right. We have all of the different tools stitched together and it's really cool the only the only problem is it's also kind of slow. We end up writing data out to disk or s3 respectively between each of these tools, and it turns out that the only thing slower than disk is distributed disk. That's not actually true. Tape is slower than distributed disk, but you know it's: it's not a fun time right.

A

um It's not able to help us turn our non-distributed tools into distributed right so like. If we have something that isn't already paralyzable, it's not going to be able to like make it a party for us. um It is going to be able to schedule it on like a really big chunky machine for us, that's kind of cool, but like it's, not it's not going to magically make pie com run on three machines at the same time.

A

So how could we make this like suck less right? There's a whole bunch of different things that we could do the first one is we can avoid writing to disk um and and technically we could do this with kubeflow pipelines, because we can return anything.

A

But this is the same sort of technical, wherein technically everything's a turing machine, so yeah you can but like no one's going to, um we could move like our parallelization stuff up the stack so right, instead of just using spark to parallelize things we could see if, whichever workflow tool we were using, gave us some ability to parallelize more things and there's probably a bunch of other things that we could do. Oh by the way. Also, this is my dog. His name is professor tim bit.

A

um For some reason, trevor left him off the paper, but he is included in the book and if you buy a copy, you can see another picture of him um right. Okay, so probably we could do more things, but, as as mentioned before, I am very lazy um and right. So so we, the other thing, is we have three standards right. We've got like these three different tools.

A

The solution is to make another tool. Does anyone remember that xkcd for copyright reasons? It's not included, but if you don't remember it, you should definitely go look it up, because that's that's what we did or another book right. The software engineering thing ah it does pay a lot more than books, but right we could.

A

We could make some some new books, or you know these these tools, so we can use ray distributed to make using multiple tools less expensive, and we can do this because ray is able to represent the data in an internal apache arrow in memory format using its object, store called plasma which has a nice shared memory interface and if any of you have used shared memory before you know by when I say nice, I mean dumpster fire but contained dumpster fire. um So it's really good really fast. um So we could.

A

We could make using these multiple tools less expensive, so we could use ray to stitch our things together, instead of just depending on cube flow pipelines.

A

We could also just try and have less tools, and we could do that by doing things in dasc and, for example, das has just distributed svd as well as having the ability to distribute the picom stuff. So we could, you know, use something like desks, so we have less tools to stitch together and even though every tool that we stitched together is still going to be kind of expensive, there's there's less of them going on, um but regardless the answer is buying more of my books.

A

Okay, so you might be saying to yourself holden. That sounds great.

A

I will, of course, buy those books that you mentioned on the previous slide, but should I still buy your kubeflow book and the answer is yes, um and so this is because rey and ask don't solve everything um specifically when it comes to the level of isolation that we're able to get um ray and ask make it really easy for us to use different versions of python libraries, but if we ever have to deal with like different versions of cuda or like different native libraries, our life is going to be really really sad and we'll want to use something like kubeflow to do the coordination instead of trying to use ray or das, which have a much tighter integration.

A

um Other things is serving um the ray people if you're watching this talk, I'm sorry! So you can do serving with rey, you probably shouldn't, um or if you do you're going to have an exciting learning opportunity which is going to give you an exciting opportunity to give a talk about all of the things that went wrong with building a serving system in rey. So you you can do it and you can fix those bugs because it's open source, so please do.

A

On the other hand, if you are in any way judged for your performance instead of hours worked. Consider not doing that and you can use something like selden for serving, which is like much happier- and I say this because I think one of the seldom people is in this room and I'm not sure if the rey people are in this room. um Okay, also hyper parameter tuning.

A

So, okay, yeah, there's a whole bunch of really lovely tools that we get with kubeflow that we don't really get with rey or desk they're sort of more designed to make these things parallelized and less focused on the fancy machine learning bits. um So that's that's the party there cool, also, if you've ever said to yourself holden. This sounds terrible. How can I get my children involved? Have I got the answer for you?

A

You too can teach kids about the magic of cube flow with cube's flow place palace what no place they should have called up palace, whatever um it is available now on the internet and, unlike all of the other books, it's free, which is clearly a mistake, but before they realize this mistake, you should go and download that pdf and show it to your children to save them from a life of software engineering.

A

um Alternatively, if you know kids, who do not have your home phone number, you could teach them the magic of spark with distributed computing for kids, which will be available for values of soon which are similar to when I will do my assignments, which is soon and you can teach them the wonders of spark and then later on. If you make the mistake of giving them your phone number, you can teach them the wonders of out of memory exceptions.

A

Yes, not me! Oh! I should take my phone number off the internet. Oh dear, okay, um anyways, so that's that's it! For for the talk. I would really love any live questions. Alternatively, if you're shy- and you don't like asking questions of the person wearing the fabulous dinosaur dress in person, um you can always email me. Thank you. One person who clapped it means the world um you can also email.

A

Trevor trevor is delightful and will answer questions about the math things, electric bike imports into chicago and whether or not he has slept oh and also also, this is totally only tangentially related. He proposed to his wife in the book that we wrote together in the in the preface, and she said yes and now they have a kid, and so I feel in a very small way, partially responsible for this, and you too can feel partially responsible by buying several copies.

A

Okay, does anyone have any questions.

A

That looks like a no oh did I go over time. Oh okay well uh I'll be around um because I am cheap and, I think, there's free drinks. um I hope I hope someone told me there were if they were lying to me. I would be very sad. I'm gonna go, try and find free things to drink and eat and you can come and find me um if there's another person wearing a dinosaur dress. Please introduce me otherwise. You can just look for me by my dinosaur, dress.

A