Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Large Scale Teaching Infrastructure with Kubernetes - Yuvi Panda, Berkeley University

Description

Large Scale Teaching Infrastructure with Kubernetes - Yuvi Panda, Berkeley University

Data Science & Programming literacy is an important aspect of literacy in the 21st century, but teaching these skills at scale is quite difficult. At UC Berkeley, we are trying - our 'Foundations of Data Science' course has no pre-requisites, and routinely attracts more than a 1000 students from across majors.

Requiring students to have local programming environments installed & debugged is a non-starter at this scale. We have been running a Kubernetes based JupyterHub environment that allows them to do all their programming with a web based environment with Jupyter Notebooks. This is an important change in many ways:

1. Lets students start instantly with writing code, rather than dealing with the accidental complexity of installing software locally

2. Acts as an equalizer - a student using a chromebook borrowed from the library has no disadvantage over someone using an expensive Macbook Pro

3. This is course critical infrastructure, and needs high availability at low human / dollar cost

In this talk we'll go over how we have:

1. Used Kubernetes to make reduce our costs while allowing a larger group of people to deploy safely to various cloud providers.

2. Extracted our JupyterHub deployment into a project part of Project Jupyter (Zero to JupyterHub) that is being adopted at other universities & organizations.

About Yuvi Panda
UC Berkeley, Data Science Education Program
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Well, thank you all for coming. There's lots of other interesting talks going on. I've is successfully tricky Olympic being here, you can collect your free money afterwards, so I'm Yui, I work for University of California at Berkeley, I'm, also part of the Jupiter project.

A

People here know about the Jupiter project. How many people achieve a project? Okay, so I mostly work with them on scaling things like lots of people and that's what the talk is gonna be about.

A

So this is a image I really like to use. This is day one of UC Berkeley's foundations of data science course. It has about thousand two hundred students in that one class, most of them they're like from large variety of backgrounds. Some of them have like some amount of computing experience, but most of them don't. But you know: they've used the computer for a long time, they're like sixteen year olds in 2017, but they've not like used the command line or anything of that sort.

A

But you know they're here to like learn about computing, so this has like a lot of challenges in how we can support those people in learning without like having to like deal with installation issues all the time. So these are some of the problems that it'll deal with. First is like setting up and supporting these many environments is a major pain in the ass.

A

If you're an instructor or a TA, you want to be spending your time on teaching students and not figuring out installation issues on Windows, XP or Windows 7 or like Linux or whatever it is disparate access to competing power is like a big problem like if you have a two thousand dollar MacBook Pro, it's much easier for you to learn and set up and do things. Then, if you have like a hand-me-down Dell from 2005 or like a Chromebook, and this is something that we take seriously and we also don't want to invent anything new.

A

We don't want to have like a berkeley, specific environment, that people learn, and then you know when the students leave the system, they learn something completely new. You want to reuse things as much as possible, don't invent our own stuff and we also don't want it to be just for berkeley like this problem that we are trying to solve, which is you know like allow teaching large groups of people, data science or like intro programming.

A

We want this to be solved in a way that everyone can use all around the world and not just in, like you know, like a really top public university that can get free money from a lot of people, but we also don't have enough time or money to actually do this properly. This whole set up that I'm gonna describe is managed by two part-time e people, one of whom is me.

A

So we have to build this in a way that scales for humans, like I, don't know, I do not want to be like a traditional academics of said man I, don't know what they really are, but I in my head, I see them like sitting and working all the time and like someone, calls up and says, I want to do this, and they're like ah this is so terrible, but fine I will be the hero and do this, so we do not want to make that happen.

A

So what this all really means is we really wanted to have like a multi cloud, because we want to run it everywhere. Arbitrary code execute environment. You just want the user to be like I, have a browser, I'm gonna go in here and I'm gonna. Do all my work. I don't have to install anything. I've used lots of browser apps in the past, and this is just one more of them except it's letting me learn how to write code and we want persistent storage with all of them.

A

This is a hard problem to solve and scale, and we've spent we've been doing this for about two years now. I only got involved in this about like end of the last year, so the talk is going to be we're going to go through how we did this for the first semester and then how we did this for the second semester and like what are the things that we learnt and how are we open source them and how other people are using them.

A

So I'm gonna talk a bit about scale, so this is spring 17 spring 2017, which is some time like from this January to this June Google generously donated us cloud credits and we ran this whole infrastructure. There was 2,000 users of this and we ran it all on Google cloud, and this was like 44% people who didn't identify as male. We had 10 plus connectors, so it was not like foundational courses, but it's like there is a legal connector.

A

So people who are doing legal stuff then like take that and then they learn data science to do things with like law texts, there's an ecology, connector. There's a child development connector lots of things going on over there, and then we had a very small class in the summer. I'll talk a little bit more about what we learn from that late and for the fall, which is right. Now we are running on Microsoft Azure Microsoft has also creations. He donated credits to us.

A

It's a larger class, we're about 3500, total users and, like again mixed up over like very different connectors and very different, like majors, that people are coming from I think this is the first time we have more people from non computer science and non stat elated majors than from computer science and started majors she's, like a big step for us and in in like a few months, we are actually gonna launch a massive online course.

A

This is the whole, like you know, take it to the world situation and we are planning for a hundred thousand users from all over the world, which is like much bigger scale. We expect about 25k to be active at a time, and we've been planning for this for like several months, and this really like is where I want to be, because I'm like oh yeah, I, want to build things that when I was fifteen and with like a to physician-based connection somewhere in southern India, this would have been very useful to me.

A

So this is like great for me and like we're running this on Google Cloud, because again, Google has graciously donated a lot of credits to us. This is also why, like the multi cloud, situation is great, because different people want to give us free money at different times, and you run in like all of them, yeah even the summer somewhere, we run on bare metal, but it was also great at hardware, so I'm going to show you a quick demo of what the student experience. Yes.

A

So this is the this is open to everyone.

A

Okay, I managed to hide my my full screen and I. Don't think I can bring it I, bought it back cool, okay right. So this is the course website for the main course. If you go there, there's a bunch of announcements.

A

There is a calendar and like this is like a normal course right, like everything has this, except if you click on this assignment, what this is going to do is I've already authenticated like it would normally at this point, ask me for my like Berkeley credentials, but this is pulling down my assignment, so all of we use Jupiter notebooks for everything. So we have like you know, so this is an assignment that a student supposed to do, and this like the notebook already has like you, know the spaces where they can write their answers.

A

It has a bunch of exposition telling them what they're doing, how it related to the class that it is took this week and the students can like immediately let go and execute code here. Jupiter books are great for pedagogy and lots of people use them because you can mix like you know your output, your code and narrative in the same document- and you know this is also persistent I- think I last time, I change this like from hello world to hello world.

A

So it's still there so I pull in my notebooks and I can do this, and this is the experience two and a half. They don't actually know anything about communities. They don't have anything about docker containers. They don't even know that they're, like you, know, running on the cloud similar to them. It's like oh yeah. This is just another Gmail Facebook, like application that I'm using just in my education system, and we have lots of other plugins. Here we have like automatic grading.

A

We have like instant feedback for test, failing I'm, not going to go into that too much in detail, but this is the workflow that we want for students and we have been having this for like a while, and this is really cool for like a lot of students who are not in CS, we have modules which basically take like four hours for like a single class. So if we had like I, think an ecology module.

A

So these are people who were like studying, ecology in some form, and we would only do a four-hour course for them in like a semester, and if you only have four hours, you cannot actually teach it if you are like spending all of the time installing stuff- and this is only possible because we are this kind of instructure and that's I- think that's pretty cool. So if I go back.

A

Oh wow, so how does this happen? So this is? We are using a bunch of technology here. One is Jupiter hub new people here in all of Jupiter hub. Okay, that's cool, so it to those who don't it lets you run multi-user Jupiter notebooks. So, instead of just installing something and like you run it on your laptop, you can set up authentication which you didn't see, but we use Google OAuth for authentication and you have lots of authentication providers that people can use and then it spawns them in some provider.

A

So the Cuban IDs is one thing you can turn it on, but you can't run it on like me, so you can turn it on just systemd just talk or whatever you want, but in our case we are using communities because I'm not going to go into like Cuba. This is awesome here because I think you're, tired of that by now- and you have a day of that to go. We also use helm everywhere for doing our deployments.

A

I will talk a little bit more about our deployment strategy later, but it's been really really useful to us to use this I'm gonna go through a very quick overview of how super habits works in our setup, at least so we have a cube, minis cluster and you're logged out. You haven't like done anything you come in and then you hit a proxy. We have a proxy server running, it could be traffic. It could be anything that supports ingress, but we also have a native nodejs based proxy running and the proxies like.

A

Okay, this user is not logged in I, don't have a route for them, so then it sends them to a hub process, which is a centralized process. That's running Python that we have and then that redirects them to the authentication flow depending on what we're configured or what does the redirect dance and then it's thoughts apart. It just talks the humidity's api and then it waits for a part to start- and you can you know we used like automatic volume, provisioning manually provisioning whatever you want, that's all configurable.

A

So we wait for that to start, and then we talk to the proxy and be like okay yeah next time the user comes in just send them directly to the part like don't go through us, so it's like hub is sort of like a Cuban ease controller, also, not directly so, and then we just read out the user to like their prefix and then and then now they're logged.

A

Then the user comes in, they hit the proxy and then they just directly go to the pod and directly connect and the hub is taken or the equation, and this lets us scale a lot more easily than if we have to have the hub in like every single request, and it also lets us like do like restart. So if the hub is down, it's not hae yet so when we do a restart to do a deployment, then users can't authenticate for about two seconds, which is much better.

A

It still sucks, but it's much better than like no being able to use the system for two seconds. So that's like a very high-level overview of how this works. I can I can tell you more about it when we catch up in person, so we we're gonna go through like a bunch of things that we set out. We want to do and how we accomplish them. This is gonna like so from the problem sections. If you remember like you know, we wanted a multi-tenant setup and we kind of did that right.

A

It was like very easy to do and at this scale, like I, find it hard to like call like 3,000 users like a massive installation. It is like it's 30 gig, it's a 30, terabyte ram cluster. Everyone gets a gig of ram, so I feel like it should be massive, but it feels kind of wrong to say that. But it's like a solvable problem. We fixed it, but then we were like now try to tackle all the social aspects of this. One of them is.

A

We do not want someone who knows and understands cubed ease to be to set up superhub. We want it to be like run by everyone if you are just like a TA who has a 60% class and you're spending. All of your time, letting people install things, we want you to be able to do this without actually having to learn all about cuban. It is so. How do we go through that? So I'm going to go through the history of how we set up so for spring 17, which is the beginning of the year.

A

We were running on google cloud and what we did was we just like you know we had like 15 days to get the set up and but we've been working on a helmet for a while. So it's not that bad.

A

We set up a google container engine as it was called then manually, and then we just you know, ran helm, install commands you're like okay, okay, let's get this running, oh wow, it works. Okay, let's go and, like finish all the other things off, and this was very useful, but it was also like very specific right. It was like a berkeley, specific installation. This is where, like a lot of academic input, deployments just end up being right. Like oh yeah, we have this cool thing, let's show you and then you're like.

A

Can we use this yeah? You can come join our university and then you might be able to use this, but otherwise it's not good enough, and then we had. We had production on staging you're not like that bad, but they were all the same cluster, just different namespaces, which was still useful. We were able to do deploys to staging and like break that into production, but setting up a new cluster was still kind of hard and summer 17. So this is a much shorter course.

A

It only had about 200 people, so we had like more time to experiment. So we had bad metal stuff. We had six nodes, they were donated to us from the hospital school who are pretty awesome. People we created like a bunch of cubm scripts and just manually went and installed stuff there. It was not repeatable any farm, but I feel like for. Like a course. There was only two months. It was totally worth it.

A

You don't have to like go through the whole setup for every time and we learnt a lot about setting up manual installations, but the really cool thing we did was. We took our Berkeley specific installation, then generalized there into an open-source project, and then what we installed on a summer was actually the open source project we didn't actually like carry over our stuff, so I'm actually gonna. This is the open source project.

A

It's called zero to Jupiter hub, it's really very nice, it's a Helen short and a bunch of documentation that people can follow step by step without entirely having to understand what they're doing. That sounds really bad, but it's been actually fairly useful in practice for a lot of use cases. So we want to target people who do not consider themselves operational. So what does this mean? This means that, like by default it's secure by default. It has all the best practices and it is not too jargony.

A

So we were able to like abstract the help chart helped here a lot, because people are able to understand. Okay, I have a config file here, I'm gonna put these things and then run them and then able to slowly learn from there instead of like ok, now you understand pods, and now you understand deployments and then, like two weeks later now, you can have a hub running. We really wanted to go from like I.

A

Don't really know anything about the cloud to like I have a running Jupiter hub in about 10 minutes, and we user tested this very heavily with people who do not customer slows like SAS admins, and then we like changed both our chart and the darks to match that, and we kind of consists like the Ubuntu for Jupiter hub, because you know like Lux color was great, but you have to install gen, 2 or Slackware or Debbie, and you had to be like this really, whatever the new boon 2 is like no, that's fine.

A

This is for everyone. You don't actually need to understand what you're doing, but you can use this for cool things that you care about, that you want to do and that's sort of the thing that we set out to do and it became part of the Jupiter project which it has the white umbrella. There's lots of projects in there and that gave us governance models and, like we were like very happy to be accepted by them and like there's a lot of community people there who got involved.

A

So it stopped being like a Berkeley, specific thing and just became like an open source project that Berkeley happened to use and that's the URL. You can also google for this. It's gonna show up so what I'm gonna do now is it's gonna. Do a very, very quick demo of how this actually works? Is that visible in the back? Yeah? Okay, so I'm gonna try to do this from memory, because I can't see my speaker notes. So what we're gonna try to do is I have a cube, custard, that's already running!

A

So if I do cube serial version, it's running 1.8. This is running on Google Cloud, so what I'm doing is I'm literally copy pasting, the commands that isn't the guide I'm not going to copy/paste, because I know them from memory and I don't want to like go back and forth so I'm gonna do having repo ad Jupiter hub has GDPs.

A

Okay, so hum repo update, we're not in the official chart repository. We have our own thing because of lots of reasons that, if you're involved in I can tell you and now I am going to create a config file, it still needs like a secret token to function. We don't want to auto-generate it because that's very insecure, but users seem to be comfortable doing just this. One thing and we say, like you know, like run open, SSL and like copy paste set, but I'm gonna use something foo to make that happen.

A

Right so we have this config file now and we basically want to say: okay, let's just realize this into an actual Jupiter hub instance. That does something that's useful, so I'm gonna do have them installed. Name equals to cube con demo. Two names space equals to cube con demo to Jupiter hub Super Hub. We tell people to version everything explicitly so 0.5, then F config toriyama. So this is also just again directly from our the zero to Jew. Prop guy. You can copy/paste this and it'll just like completely work.

A

It's gonna take a minute because we also pre pull all our images because we they're like sometimes really large, and we don't want our users to like wait for them to happen when they first show up okay. So this is now setup and I'm. Just gonna wait for like the service to what I call this cube. Demo.

A

Okay, get us we see and it's the pending.

A

Well, that's gonna take a minute, but that's it. So this is pretty much all of it and, like the deforestation, is already useful it ships with Condor it ships with a bunch of like useful base, libraries that a lot of people use and, of course you can switch it out for your own image. If you want very easily by just changing the config files, we also give every one: every user gets a gigabyte of personal storage by default when they log in and we use cuban ids is really nice dynamic provisioning to make that happen.

A

Okay, I have a external IP now so I'm gonna go back to a different browser and I'm gonna open that, and hopefully that works yay.

A

So by default, the Authenticator just lets you in no matter what you have so I'm, just going to say, cube con and okay type, for it doesn't matter, but you can easily configure this to use whatever authentication you want LDAP or what and whatever else? Okay.

A

So you do this, so this like creates a persistent disk and attaches it to your pod. So that can take several seconds, but so this, like a lot of people, use this for their workshops. So if you have like a day-long workshop a lot of times, it just ends up like you're spending time installation installing things now.

A

Instead, they just set this up their day before and they you say that, like the free credits on Google cloud or like, we also support instructions on Amazon, Azure and some of the academic clouds, so people just end up using whatever it is that they have and the multi cloud nature of this is very helpful because we don't actually have to like set up a separate installation set up for all of them.

A

We just like we'll get you to Cuban ideas using any of these external solutions and then, on top of it, it's all kind of the same. So.

A

Well, usually, it's not as low.

A

The disk provisioning sometimes ends up taking way too much time. I'll talk a bit about storage. Afterwards, it's been really the biggest problem for us yay we have this, and users can also like distribute materials with it. We have instructions on how to do that. So a lot of workshops just have an initial set of materials. Then they set this up and then they do this and they tear it down, and it's very useful and the people are just like.

A

Oh yeah I just want to teach people like you know like Piton short, like psych it and I, don't actually want to learn about Kuban, it is, and this dust-up it was very well well also like for people like me with learning large classes. We want this to work for us too, so it has all the knobs and whistles that we can turn to manage like a large class like with automatic deployment and whatnot, but it also is useful for, like the other use or the other persona, all right, so there was the demo.

A

It actually went pretty well right. So this is like we call it a Democrat izing distributed application. Department I don't actually have to understand what is going on to be able to use it I think that's the same thing with like all of the apps like I have install it. I, don't actually know what's going on inside.

A

Someone else takes care of it for me, and we've been very happy with the turnout from here right: okay, I'm gonna go back to talking more about like our continuing set up so for fall, which is happening right now, we're running on Azure, and this time we fully automated our turn up. We started using NFS for storage.

A

Unfortunately, I'll go into detail about that later, but instead of manually creating it, we wrote a script that turned it up for us, like I've heard a lot about like oh yeah, you should automate your class or not, but I'm like that's, not doesn't seem worth it like. You know it's gonna, take me two hours to do and like I, don't know, I might do it once every six months or once a year.

A

It's is it worth it, but it totally ended up being because you were able to like very easily performance test, multiple things like because, instead of taking two are set up and then we are not sure if the setup that we did was free producible, it's like two minutes is set up on like a script, so it could be like okay, let's do this and do this and then test it out, and that was very, very very useful.

A

We were able to fix a lot of performance and like race conditions in our application, because we were like okay, let's just finish our and throw like five thousand similar similar to users at it, and that, like rats out all the race conditions- and we did that for like like almost a month to make that happen, that's all because we were able to automate cluster turn-up, and that was very, very cool.

A

This also, let us have fully separated State in the production of Brahmins, because you know that was like one line instead of like okay, I would spend another another day, doing the exact same thing. No, and now this is the online course which is like an order of magnitude bigger right. It's like hundred thousand users versus two thousand final users, which is totally different. So this time we have like completely automated everything.

A

Everything, including like provisioning of the NFS server, setting up multiple hubs that we get load balanced against and last time we had one NFS server. This is gonna, be 20, NFS, servers and they're.

A

All totally works are ok and we have a simple Python script that drives both helm and Google deployment manager, cool deployment manager is, like you know, AWS CloudFormation or as yours, like as your resource manager, and it basically has Yammer that lets us Nick you let you specify the resources that we want and it's it's very cool, and it also has like we fully separated them, including in like separate projects, so that you know like we don't run out of credits and one or the other right.

A

So that was like that's where the MOOC is and like I've, very happy with where we have come to along there. So the next thing that we want to want. We want teaching staff to be able to do most of the deployments themselves. We don't want to get a call saying like oh my classes at 9:00 a.m. I know it's only 8 a.m. it's 8 a.m. right now. But can you install this for me because otherwise, my class is not gonna run.

A

We wanted to put that power back in their hands so that they can stop bugging us, and so we did a bunch of work around automated deployments. One is, of course, where do we store our configuration because it's a Yama file, of course, but like? Where does it go for spring? We had a public key debrief. Oh I come from Wikimedia so, like my instinct is always like. Oh yeah, everything should be public by default unless there is a very good reason. It should not be so.

A

We had a public at every per with all of our config and we had a private repo on like a single Bastion server that we backed up. Hopefully, that had all the private config and we just merged them together when we did stuff- and this was ok- I'll talk more about the problems that we run into after that and for summer we were like this is fine. We just have a private repo. It's ok like nobody could touch this it'll just finish up for fall. What we did was we completely like read at this.

A

We started using a public key W power, and then we used to get crypt. How many people here know about get crypt? Ok, it's a very, very cool and useful tool that I'm sure it's gonna bite us in the ass at some point, because there's no way if this could actually be really like this. So what it lets you do is you can use either GPG as symmetric key that lets you encrypt certain files in your git repository.

A

So if you have the key, it looks like an unencrypted file to you and it transparently encrypts it when you push it out. So we use this for, like Oh our secrets, cookie secrets, like various tokens, that need to talk to each other or things like just student user lists, sometimes because you know that's protected data with some regulations and whatnot and then we automatically deploy from Travis.

A

So if you there's a if there is a security issue and it crypt or Travis kind of screwed, but hopefully I think so or so many of the projects so I'm like okay, that's fine, the world will become bling it on me for I made. You worry about this particular one and we were doing the same thing for the MOOC, and this is very, very. Very nice I'm gonna talk about our deployment workflow next, and that will make it work here. Hopefully, so this is for the spring 17b.

A

So for the for us, if you are a if you're a TA and you wanted a new package, what do you do? Is you would go? Make a PR on our github repo and, like just add, to the docker file? Okay, I want his new thing and then someone would code review. It might be another TA.

A

We ended up everything, a good community of these people who are doing this and it'll run your tests to make sure it builds, and then they would merge on slack sorry, merge it on github and they'd have to show up on slack and say: hey I'm doing it apply. Nobody do anything, please let me have this and then they would have to search in and then run commands on a Bastion. How many people have done a deployment situation? That's like this okay.

A

So this was not bad like the PR. Let us do code review like people didn't like do obviously broken situations, and we had a staging so that caught most of these, but we also had situation with someone like I'm doing it, deploy and then they'd forget to say that they finish doing the deploy and then they would go home and we would be like like what this is like a team like TFS or SAS, VN days, where someone's locked something and it deploys I, never felt safe.

A

I once destroyed, like everyone's data by doing deployment wrong because of get some modules and thankfully was before class started. So you're, okay, but it never really felt safe enough that I could we could like completely let the students do it entire time for somewhere we were just like you know it works, let's not touch it. It's only two months. Let's move on to do something else: I mean should be dead for fall. We spend time making this much better. This is now a proper like CI, CD, environment yeah. You merge on github.

A

It will automatically deploy from Travis using git crypt for all the secrets and we were able to do helm upgrades. So if, like oh, this class needs four gigs of RAM for the next week. Instead of two or like you know, you've got an image update, that's fine! The students are able to completely do that without any input from us, and that was really cool and so most PRS are now like self merged.

A

We, we feel confident enough that, like you know, like I'm, not gonna, look at your PR until you explicitly asked me to, and you can just submerge it and test it on staging and then merge it to production and that totally works. We have more about 150 merged PRS on this, and it's like very empowering for them to, because I don't have to go at their speed. They don't have to go at my speed. I, don't have to be annoyed at them.

A

That they're interrupting me and they're not be invited me because I am not paying any attention to them for the more we are doing something very similar, except instead of just helm updates. We are also going to do full-on Google updates.

A

So if you are like I want a new cluster, because you're running out of capacity or I want an entirely new hub with a different configuration because I'm teaching a separate class, you just change the Yammer file and then merge on github and will automatically deploy, which is different from what we have right now, because we don't create, like you know, like the we don't we haven't.

A

We have automated the scripting of creating new clusters, but that's not like automatically deployed in any form, and this is this is really nice because we are like expand and contract. We might. We might we're not we're not gonna like provision 400,000 users in wait. We gonna make provision for 20 day and like okay, that's good and then keep bumping up until we get to where we want to go right. So this is great, like we were able to like spread out a response, please for deployment and that helped a lot for morale for everyone.

A

Alright, the last bit of this is about storage, I, hate storage, because so, like the worst thing that could happen, is students lose their homework right, like the computer hate my ate, my homework should be like an excuse. That's not actually true, and if it is true, then that's kind of really bad and we should never do it, and it also needs to be cheap, because we don't have that much money and I wanted to be reliable. I don't want one student who accidentally wrote like infinite loop.

A

That hid your disk all the time to like screw. Everyone else, I wanted to only screw that so in spring 17. We almost had all of these things. We we had one google person, this first student, so whenever the pod would get heated, we'd have a PVC that will dynamically provision them, and that was great. We gave them all SSDs and everyone was really happy with that, and you know, like all of these things are handled for us by Google, like you know, that's their provider and they provided this cool infrastructure performance.

A

Isolation like if one student takes up all that IO. Nobody else is affected back up. We just use like volume, snapshots and I was good enough and Kota as well. You know if you go or 10 gigs, it's you can't, but it was so expensive once we had auto scaling like the storage cost was about 3 to 4 X or our compute costs, because it's super inelastic. If we pay for 10 gigs of SSD and if the students 90% of them were using only one gig, it's it's just very, very expensive.

A

So we had to stop using this. What we really wanted was to be able to over provision. We want to be able to say, like you, have a 10 gig quarter, but we expect most people to use only one gig, so you're gonna provision for about a gig per student and whenever someone goes too much over it, we're gonna go find out and like SWAT them or like send a scary email from their professor or something like that, rather than do anything much more drastic.

A

So for summer 17 we were like- let's not solve this, so we use the incident like it was running on bare metal from like a department that already had an NFS server, and so we used there's an NFS volume provider in that is entry in community. So we just use that it's only 200, so it's fine but is still problematic because it was doing one NFS mount per pod, which meant if we had 100 users running, we had 100 NFS mounts and when the NFS server goes away at this.

A

At this point we had then go clean up all of them. Some of them aren't stuck state and that's very, very nasty, and we didn't really want to do that. We had to do that a couple of times and performance. Isolation was just like. Oh it's someone else's problem, and you know like the asari book has like a really nice code, which is hope, is not a strategy, but in our case it totally worked out.

A

Ok for fall, we were like ok, we can't really do like the single disk per user anymore, so we had our own NFS server and this time, instead of doing the NFS a Wally mount. What we did was we mounted the NFS share.

A

We had one server that was exporting one share, so we just mounted it on all of the nodes and then we used host path to actually like get that content into the users part- and this was much better because, instead of having one Mountford part is one month per node and when it screwed up, you know we had 30 nodes, so we would just have like SSH in and like clean up all of them. This is kind of nasty, but okay, and so it also acquired out of out of band work.

A

So we're not like a you're, not pure Cuban ideas, anymore right, like we had an outer band and FS server, and then we had to like had something out of band on each of the nodes to mount the NFS, because daemon sets were not cool enough to do that. Yet this is one point six or whatever, and that was not very nice, but it's definitely better than just mounting like 800 or 2,000 and FS mounts and again performance isolation was just like: let's just make it very big and they totally worked out.

A

Okay for the MOOC we're doing. We have a lot of animal service when FSF is not gonna be good enough, and so we didn't want to do out of bandwidth. But thankfully, with 1.8 flex, volumes became good enough that we could actually start using them. So you wrote a custom flex volume provider that does NFS in a little differently than the inbuilt provider, and it works pretty good. It does one mount per pod and then does a symlink for mounting it to the volumes and it also automatically like remount set.

A

If you run into like still file adults or whatever- and that's that's been- that's been very good for us- we've been running tests with it and yeah ansible ZFS for the NFS seems to work okay and performance. Isolation, we're doing this with TC, which is traffic control.

A

So you can like limit the network traffic to the NFS servers for putting out isolation, which is still like not as good as per pod, but it's much better than not having any and I picked up this trick at Wikipedia, which is pretty nice so enough, it sucks, but we don't have enough people to run blast. Ref s I feel like it's not like a part time situation that someone can run and I don't even know if it does like over-provisioning blast of SF any of them, and it totally fits our use case.

A

Nfs is like yeah home directories and we don't really care about the performance. We're not running a database on it's good enough and we have lots of, like you, know: academics putting it in the face for a long time, so yeah someone, please fix this. For us. It can't be us, but I, think lots of people are working on it and that's great okay. So what are we have yet?

A

So this is great that was supposed to come in one line at a time, so Cuba Diez works very well in practice, like we kind of very able to take it for granted that once we had committees up and running on platform it we can just do we don't have to repeat all our work like getting companies up and running was the work, and then we just like start from where we, where last semester or the previous time rather than start from scratch, is super cool automatic lassen turn up is great.

A

This might be like you might already know this, but I was kept achill and then once I did it was like I'm, never going back. Democratic deployments pays off if we do not want to be a gate, we want to have guardrails and we do have them and they're like are like star fans through teachers like really love us, and it's really nice distributed storage, still sucks and distributing secrets still sucks and I. Don't think.

A

That's news to anyone, but if you have a solution that for all of this and this fully open source come talk to us, so you can be part of this. This is all open source projects. The Jupiter community is awesome, there's the helm, chart and the documentation for it. Is it zero-g, perhaps I'm not going to put URLs there? You can always google for this and find out and the Cubans integration is is called cube. Spawner.

A

That's the Jupiter hub plug-in that talks to committees and makes all of this happen, and these are some of the things that we're working on we're going to like scale. Two hundred thousand users we're doing lots of a phone testing and we are working with some of the people from Google on Lake tensorflow integration.

A

We working some of the people from dusk on on again like working with tasks and we're also working on CI for our hand, charts because it's a fairly complex um chart, we wanna figure out what to do and that's our like jitter, which is what we use instead of IRC or slack, and this is all not just me. There is a massive team of people who I'm not gonna list but they're all part of the chipra team, superhub team.

A

They are awesome, so this is a very great way to do things that are both technically challenging and have lots of social impact while working with a pretty cool theme. So that's it. Thank you all for coming. I, don't know. If you have time for questions, do you have time for questions? No okay, but I'm gonna be here. You can find me with my hair. Please feed would ask me questions.