Add a meeting Rate this page

A

But if you want to.

B

You.

C

One more minute: okay,.

C

Another some people I don't recognize, um grant Nicholas Jean from school, underneath hi.

D

Yeah, so it's great Frances uh from the air for community he's a grant in Lincoln Mitchell months and our core it with the airflow and proving that he's initiatives.

E

Okay,.

E

Was.

D

It like difficulty: okay, I shuffle.

F

Chloe me I'm from a company called life scale. We make storage segregation for for the bear, I.

C

Missed the last our storage segregation for what the segregation three.

F

Greens, aggregation not segregation; okay, okay, actually, I can write it down Columbus, it's very easy. You extreme like! Essentially, if you look at a regular coastal, you you're using dry, cl' attached storage, or we provide like creo student government for direct attached storage, but you can dynamic on the line comment. How many drives you walk on a given server? Oh it's.

G

A storage virtualization then, when we say disaggregation, because I.

F

Write it on apply and okay, just let me open that.

H

Yeah, we are definitely a walk you up and please get into the school.

F

So yeah we're working on the on the Flex volume plug-in and we have a walk-in clinic plugin for criminals and we're working on clinics seeking at the things to to get HDFS to work quickly. On top of that, in terms of locality and failure, domain.

H

Okay, yeah.

C

So knock you get enough. Announcements is most you already know. Of course, we finally dropped the sparked 2.3 rebase release and yes tip for getting it adopted upstream. Hopefully, this release cycles, so that was a big deal um and thanks to every to rule for all the work, you guys put it on doing that, and people well spent a lot of time on mouths and the lock will be able to see the fruits of that library sometime between now and the February believe yeah.

I

I spoke briefly with Renault, and he said he would talk to the other PMC members about getting us support on that side, but to all the people that came the post if you can go, and you have your plus-one and the reason why this should be up streamed. That would be super helpful, I think those people here have already done that. But just you know there and again has except thank you for all the efforts around dogs. The you know, diagrams.

B

And.

I

The design, doc and everything so hopefully this I haven't seen any minus one or anything that seems to indicate that people might not want it. So a good good signs all together, but we'll wait for the PMZ votes before we conclude hoping.

C

I notice that there have been at least a few totally unfamiliar people Kinney adding +1. So it's really good sign and I, don't really like discussing without legal and stuff. What you can talk about in terms of the deployment sizes but I think anybody anybody wants to like discuss to whatever degree of detail that allowed like the use cases that they've exercises on there definitely will help and as I recall, honoree of all Reynolds Pacific, Way, Africa, I, think yeah.

I

I think it's just signals for the upstream community to judge how important this is and how much you know, support it might see in terms of maintainer ship in a long term.

C

Yeah one of our close with the moment yeah.

D

Anirudh, just wondering is there any chance, you could try, sharing your whole screen and then just making the Google Doc upholstery. Your background is still like clicking through in like random spurts.

I

Erickson earring up.

C

Mine husband, hang on cooking I, laughed up and seemed to handle the conflict of two monomers very well all and where we figured that up. I.

C

Just don't hold it as it's sharing now I don't get. A we share is.

J

It's not better, it's kind of better just now, yeah, okay,.

C

Or should we figure out what the problem was? Okay,.

C

Yeah, so what will be the? What I wanted to discuss of those two topics and air flow? Has some discussions so he's going to be dangling a merman, we're going to be like leaving some uh-huh no more discussion. Time of all.

B

The airflow.

C

So we're going to like reserve time for that towards the end make sure he can. You know address these topics.

C

So I don't know we just had a big load on the release. Maybe there's not too many of TRS yet, but Matt dropped the PR for doing the specifics mounting on small piles, I.

C

Guess me: I, haven't put an LG TM on yeah I thought I was going through it I, don't see anything problematic with it. I noticed it was failing integration testing for some reason, I was hoping. We didn't get a another integration testing problem. That's.

K

A legitimate, that's just less a legitimate failure, because the copy command is pretty incorrect on the developer, so I, but.

I

Matt we're going to quickly explain like what the PR does and like just for the people who are unfamiliar yeah.

K

Sure so the use case is basically that sale 40 go for an application from yarn you, but you end the migration. You make all of your jars into your docker images, but yours still want to use smart dot files, basically, like the add files feature to ship over this trip over like some files- and in this case these files are very, very small and it seems a little heavyweight to have to install a resource staging server and also have to use a and youth in an entertainer in this case. So what this PR does is.

K

It adds an alternative mode for mounting only file. It's gone up. If this jars and like this thing will work, ah but if you, if you only have files to add and there under I, believe it's ten thousand bytes as open case, and it will instead of using a I, absolutely ten dozen cubed listed so like all files total so like the under that under that amount, the text said the files are small enough. Then it will use a secret amount added files include the driver and include the executor and yeah the other.

K

Does uh it's more of a use case specifically for specifically for those who are before over applications from yarn, and this is like I, just more distinct? This is up. This is a distinct use case from adding our preferred volumes.

K

It's where it's a Grove area that I very, very specific scenario, but one that's used to be valid or doorless.

L

Okay, basically we're looking to optimize the launch time for the jobs, and this is one of the best ways to do it. Based on our profiling.

J

This sounds reasonable. I am one question, so are we creating one secret for each file or are we just about putting much both files in a single secret.

K

If we create a single secret for the entire file, that's our profile. Okay, these are you, create a single secret for every file. I've already had higher for all of the files that you're writing is put into one picker object and we're going to. We just laid out the keys, like specifics, just a file names, and then we just not the same directly. Okay, this.

J

Is good.

K

To.

J

Me I was worried a little bit about like creating too many secret files, but I guess the design Agusta cool would be.

K

Because, presumably.

G

Because.

K

We are we we limit for total size of all files to be under ten thousand. That means that it that all of the files should be able to fit in one secret should we find their clothes in there. Probably that's an Ontario correct that he could probably I should probably change it. So it's totalize after after basically for encoding right now, I think it's total size. The four basic super encoding I guess it's more correct to be after bases for encoding, though you.

C

Better recall correctly, the formal limiters like more like a megabyte yeah.

K

Right yeah.

C

But.

I

These files are likely going to get encrypted in its CD I mean granted it's not going to insulate that much I still found. It would be better to keep it on the lower side in case there are lots of these because storing like a thousand one megabyte junction at CD completely kills it seen that already.

K

I decide I thought it'd be enforced before or after basic super encoding. Do you think.

I

I think the difference should be minimal, but this yeah a factor would be more like it would be safer right. So the.

K

Powers of the phone would do it after that. It's not immediately clear to the user. How like what do I like how much they have to reduce your size if it goes over, I, don't know, whereas if we calculate strictly for before then like we just say like your trouble size was this and yeah okay to do this.

C

That should work right because then we're way under the actual women.

K

If you do any check from before, I think basically we're including inflates it, though they're.

C

Calling it inflates it, but not so much so I guess we're like 10% or 1% of the actual on occasion, I mean it shouldn't. Even if you inflate it, it shouldn't inflate so much that you're gonna pull it out. Okay, okay, yeah.

K

I think three I think three or four is fine. I think dating before I think checking circuit over for encoding inside a.

I

64 would be every three much turn into four bytes, which means right.

J

Right: one-third one-third increase, yeah.

K

Yeah, not the worst in the world. Yes, oh that's, efficient.

I

uh Okay, I guess your condition.

K

I'm running into a problem right now we're so with the way smart dog files works. Is that and it is actually exactly a problem regardless of use register or state server or not Prez basic speaking, implementation also has problem smart contract if that files, either through sports out files, are added to the working directory they do have to be in the working directory. Do like the driver needs that your processor, so that you basically know where to find them afterwards. uh That's because this is a way to do that, but the problem is right.

K

Now we set the working directory to be just market installation directory, so I guess, theoretically, at a file called, say, item file called spark batch class and then or that it have only better. A better example to get is how you add a file, that's like equal to the names we want to indirectly prepare and then, like you copy, and then you possibly add you put the file not in the working directory button to do CPE and then like source destination.

K

But if this H is a directory, then I mean you can end up like putting files in like the wrong place or end up over writing files entirely. So I need to change the working directory to be an empty directory. I tried making the files not over rideable, but soccer fire doctor like file. Permissions, is kind of weird plus cost.

K

The fact that you're running a plus effect that you can end up running the images as route so I think most bets are off in terms of in terms of like file, attributes and never religious, so I'm, probably just going to take your working directory to be an empty directory and like this from there.

M

um When you say empty directory, is that like empty dirt,.

K

Volume I like this to dr. image that the like part of the docker command can just be like, am caterer some to some location and then and then just operate on it and then to make it work near a bad thing. Yeah.

I

Beautiful is the working directory, an option. You consider the.

K

Orga directory is Kurt. The working directory is per the only positive docker image. ah So what we don't allow configuring the working directory right now from smartphones parks, wet I, know you can put the working directory inside the pods back. I decided to take this like what we that we don't expose it as an option right now. I think it's! Okay! If you don't yet.

J

There's one minor follow path and I have so if you actually don't use empty turn and then just use a you know, whatever directory that's available inside darker image, you ends up touching files in the archive storage, back-end like overlay, FS, o FS or whatever right and depending on which storage back and you choose. You.

B

Could be actually.

J

Pretty slow because darker does is a copy on write operation on this overlay filesystem, so I'm wondering actually just using empty during directory, is a better option going forward.

K

That ideal, how bad, how bad is it if you are how bad is it if we're just working with small files in jars, just.

J

Another.

K

Good question.

J

I think it becomes pretty bad when you have ongoing small updates to this file, as opposed to your copied on at once, and then don't touch it so maybe for using the dock on yes, Jason might be just fine, because it's something to keep in mind. I think you.

K

Yeah I, don't I, don't know how much stuff mark actually writes to the current working directory. I, don't think very.

B

Much.

K

I think it does all the court contemporary directors which we have so we have to work out a solution for making the temporary directories not that Bobby's right now. I, don't think we'll do that. So we should probably probably do that and but a button footage this kind. You wear this particular use case I, don't think I just study the working directory so mannequins we find okay.

J

What.

K

About the tonal.

J

Software service inside the computer, we actually use a knock-back storage back-end for that or we use a empty turn. Yeah.

K

Unfortunately, I don't think we use I, don't think we use an editor for that. I think we're just using whatever.

B

Doctor, as so.

K

Which are probably.

B

Admit all physical constants.

K

Get accessorize it was early. We should look at that. I might be wrong honor. It do. We believe mount integer since your to our shuttle service didn't thinking I, don't.

I

Remember, seeing an empty tur, no I mean infinite executing. Then we mount Coast paths which is not affected by all of this, but about it. No no I, don't remember seeing any monster could be wrong right.

K

These are all software services, a host cloud use a whole track volume, which is very different.

I

ah But it sounds like we should be writing into an integer volume like a moon set. So maybe this deserves more investigation. Can someone write up an issue to track this? Oh.

J

Yeah.

J

Sure.

C

Our problems with empty turn, volume is having style image. Anything like that or.

I

um Not really and I mean it I, don't think we would have done it. The sightless would be the same constraints that you have on the existing set up, and it is its lifetime, is managed by by the system itself, so as I think it would be upgrade from where we're at.

C

Oh.

I

I'll add one more question about the secret amount. Small files like the secrets lifetime, is that managed by the same owner reference process that we do for the other other stuff yeah.

K

Okay, so hard look I also, you want to call.

I

I guess he isn't here, but in any case, I wanted to quickly get some feedback on what you guys, think of enabling client mode, and so I tried this out. Actually the computer thing seems to well to point to. It seems to have some issues, but mostly just works where we can use dynamic allocation and all of that, so it's really nice to have. But yeah I was asking Saahil if he had looked at like Jupiter hub and trying to get that to work rather than single Jupiter instances.

I

But yeah I just wanted to get general talks on how bad would it be to enable plank mode in this way, we're in cluster client mode is to possible and how the cluster is disabled.

K

Well, I commented I commented on the pull request. I think the current way that client mode is implemented is not exactly is required. Some work. ah Currently it's still using a submission, quiet and that's incorrect supply motion should be running the scheduler backend directly inside this works with my process and that's not currently being done. So we need to make sure that that tweak, that's actually what is done.

K

ah That's tricky, because the scheduling back end is pretty opinion is currently opinionated towards having certain like a big property set by the submission client which deploys it in cluster mode. So we need to have it's a bit like right now we have like kubernetes cluster scheduler back-end, you might need you have a pretty high scheduler back-end. This is design a logistic yarn. Has this exact exact division?

K

ah So we should follow and- and we could share this- we can share like the fare components and they like run a parent like Googler to these scheduler back.

L

In again,.

K

This analogous er ah what we need yeah. So that's that's like that's like my thoughts there. um If that works and you're within a cluster or like grooves you're within a pod in, like my I imagine a it could be something like you're just within a pond, and you started like your own travel process. However, you wanted and then you just create a single spark context could burn any quick. You forbidding me timer and that should just create, like this controller back-end like immediately and yet is it that should just work.

K

Yeah.

N

This is Tristan from clutter and unfortunately, only have five minutes. I have a meeting coming up but and I don't think the heel is on the call. Who is really the one who's look at this closely, but just so I understand and I have not looked at the code in depth so, but just so I understand the issue exactly Oh. What is the problem if this is creating? So what what's the problem I mean when I saw the demo of this? At least it worked as I would have expect from an end user perspective.

N

I didn't look at underneath the hood though, but I guess you know what is what is the underlying problem? The.

K

Problem is that you're we creating a you're creating a driver pod when, in fact, that pod that you're running it I should be the driver pod itself, so like I, say I logic, I want to have a process and I say run my spark application and like performed within that pot, I, say I want to start my spark application right now. I understand, I, understand it, based on based on some of the based on some of the comments that were written and based on understanding the code it will.

K

It will create a well for another process that creates another pod. ah That then runs that then, once the driver- and that seems like a little bit see it seems like actually needs to have like both both pods in that case right. So.

N

I can talk to see him about that. I haven't seen that but I'm you know I'm basing that on you know he showed me a demo where he watched you know, keep CTL get pods and I saw some executors start up and I know it looked to me like the driver.

N

You know the he was running Jupiter, you know using importing PI spark and I would assume that the driver would have to run in that process where Jupiter is running or where the Python kernel is running so you're suggesting that there might be an extraneous pod being created, that's kind of like a pseudo wrecker pod or something somehow. The code is kicking off a driver as well I. Think.

K

So I think I think so I mean. Let me see the specifics later.

I

Yesterday and it doesn't create a new pot, it's I think it's just running it within the Jupiter kernel is running the cluster scheduling back in there, but.

K

Is it as such, because on the comment, the comment that's written here says that it eventually calls or that apache does spark. The deployer could release a client, and that is the submission client and that's the bridging client, definitely great the second pod. So, whatever.

J

It run imaging our zombie, some kind yeah, your piece using our submission klein it'll, definitely create a driver card. I think, and.

N

It will only create a driver pod if it's in client mode, if that option is sent no.

K

It right right now the way that this, the way that the way that this client mode is implemented is by calling or dot apache dust particles or disabilities are fine, and that class is a classic raiser plot for a driver. Okay,.

B

So.

K

That seems that seems, Ingram conservation and maybe I got actually get mad.

I

Observe and I can go back and double-check and fairly sure didn't create a new driver. Yeah.

N

That was my certainly impressive, live again about. This was me as the end user. It did exactly what I expected as an end user. You know what I could serve now. Things could be, you know, created really fast and stop, and you know there could be something weird going on, but as an end user, it looked like it was doing exactly expected.

N

I talked ously he'll about that and try to clarify that confusion, and you know see if we can resolve it on the Caleb issue. Yeah.

K

Okay, yeah, basically don't. Basically we don't want to be calling dispatch. This submission client class. We just want to create a schedule go back in and if we just and if we can get back to work correctly, then, while offline mode, basically for free.

N

Let's see, okay, any.

I

Other takeaways yeah just this, we don't want to create a separate driver, pod inclined mode and, if they're not, then that's right and you Friday.

A

Yes,.

I

Perfectly.

C

We don't really long go through the service of pong honors yeah yeah, which is different but similar. So.

N

You don't want to go so I got the two things you want right, one. We don't want to create this extraneous Claude, but the oak. We also don't want to go through that class. You say yeah.

K

Basically, like I should never be able chat all in this process, so we should be hated. We should do some like code trace micro tracing in the IDE and make sure we're not doing that. It's pretty tricky, because sports limit invoke instantiates like the classic the trial classes, whether legal mission, client class or the main class of the user and like eveni our client question, was it saturated reflectively. So it's kind of hard to trace down like where, like whether or not we actually make it I.

N

See I will I will I'll chat with the deal about that we can clarify with them and a.

J

Little follow.

N

Up on the on.

J

The right on.

N

My er, but.

J

You print do a typical get pass and just quote patch will create an extra driver part and the driver car do is show up in the clip. Coder get pass right, yeah.

I

Not yes, what I would expect that it didn't so there's clearly something funny getting that's sweet in this again. Alright,.

N

Thanks, thanks for feedback, I totally need to drop from a call but appreciate it will follow up on mine thanks thanks.

C

So I just wanted to drop a comment, and maybe I should actually just file an issue this, what's going through the release process there, the 2.2, Scott, then I realized, and we can feel like an all-clear. What command flag. We should use to build early polls to make them pass a little sting and stuff like that. So I got something that basically eventually passed or that reported the command labels on the ticket.

C

But we should I, don't know II think we can maybe add a little doc page or something if you want to do a room old, here's how you can do it.

K

Andrew, can you share what we've been used because I we've we've also been building I thought week, I think we've also been releasing these I think we handle some ones released in the past. If you get you, can you post a command that you use, maybe to maybe do it maybe to the get up issue or jogging stock set like the Eric? Is it yep.

L

Yeah I have a script that runs through a bunch of things. It'll need some slight tweaks, because we've used in addition to forking spark. We also afford to do so. It runs the pouncer version of it as well make a couple small changes, but I can collect script if you're very easily convertible. Sorry.

K

I thought we did I thought we I thought we did. We had a script for just about you. Work on creative stuff, we've done really scandal relating are part of estradiol. Forty right.

L

Not the doctor images part, no I, think cut up a card. I think on a road is done that in the past the.

N

Doctor images are not.

C

So hard I mean you just know: there's actually describes when the monocot alike produced, so this Marge was kind of making sure you've got a really filled with all the rightful spoils enabled I think that was issue yeah.

K

So just say: I just yeah I'll find a rifle cause. What we yeah we can. We have a rifle context there as to what we do. Okay,.

H

Yeah.

I

Try to walk someplace in the F Docs, maybe would be a good idea just to avoid rework the next time.

E

Cool, that's.

K

Also important, to put it there to make sure we're it's also important. To put it estimation, work is just another cost built. We don't want to have I've in our first iteration and then we lose it somehow. Yep.

O

No.

K

Important.

C

Cool um I guess we move on to HDFS, then I'll all copy on us. You and I right. Oh.

P

Yes, so um I guess four four and four I'm running out the universe that otherwise, when we should be looking when migrating to two points to accretion cool um the only goal. One thing that I found kind of odd was I was trying to run two different tests on top of each other, one which runs on a K in it.

P

While reading key tab sleep ass into summation five and one that requires commissioning the keeps has insufficient client arm and the first time, I run it with the key tabs everything passes and then, when the precision volumes are taken down with the second run within the same namespace, sorry within a different name space, even the KDC note is saying that um that the key tab that was created was ordered. That already exists.

P

So it's almost like the person's volume didn't actually approach to all the combines and it's being destroyed, which I thought was quite weird, I, don't know if there was ever a case where, after you do dot deleted volumes that persists anyway, what kind of persistent volume it is it I did it on host path? Call me quickly check the gamma five, its.

I

Host pad there is no cleanup per se, so if you were to attach back to that same path, it's going to stick around on the node. So till we have precision local storage, yeah host Pat volumes are not going to purge it for you, if you want to use something, that's managed like send an empty door, is the better choice.

J

A good input, but this is a only specific, the universal past right. Fortunately, so we reuse this. The persistent vol 42 file only for the innovation test, that's a real deployed environment- does not rely on this ship.

B

And persistent.

J

Logic desk: oh okay, yeah.

P

That's interesting, I. The whole class thing is yeah. Okay, that's a good point. I'll! Look into that! That's the only thing besides.

B

That everything.

P

Passes in the end, Jenkins finished yay hi.

J

Yeah, that's awesome. I have one question so the we we use the privileged secret for mounting the delegation tokens and understand. We have a post-test for creating a secret on-the-fly to redo anything, to destroy them when oh great, the driver is done. Oh.

P

Hidden move the secret I believe in our stock. We actually specified how the cleanup process needs to be handled by the driver Paula to delete the secrets on completion tasks so.

J

All here, when you are discussing the small file secret in a PR on us, mentioned that we could use a corner reference to automatically do it. This is great onions. Can you give us some more context about that sure.

I

So if you add an owner reference from one kubernetes resource to another, when the owner dies or is you know, deleted from the API there's the pod or actually there's a garbage collector and inside kinetic, that would take care of cleaning up all the dependence on that resource. So right now we use that within the driver, part like all the executor pods have owner references, pointing back to the driver and the secrets that Matt was talking about for mounting files.

I

Those will also have oh, no references pointing back to the driver, but this the flow that exists outside I'm thinking so I'm, not sure exactly how we can get the owner right.

J

To.

I

Talk to do.

J

You actually have a 20 front code paths if I'm correct. Oh, you know, one code path is where the driver actually creates the secret file and own it or the lifetime of it. There's a the other foot path is, of course, a pre-existing secret. The user is applied to the driver right. So for the second case, you didn't.

B

Want.

J

The secret to be destroyed and the driver path goes down, but for the first occasion where the driver is actually creates, the secret we actually want to destroy in the driver path goes down.

M

Should I.

J

Think.

C

We can use on the reference for.

J

The first case yep.

C

Why are we using host Towson in, like I, say kind of problem, eventually problematic, you're trying to like have to physically physically tied by the actual nodes right.

P

Yeah three: that's kind of a hack.

I

Exactly for the integration test, yeah.

P

That's for the integration test form.

I

Okay, yeah, it's just for the integration test is for leaks. Finally, it's there's nothing that mandates users using host pads in a production, Channel, correct, yeah,.

P

Oh yeah yeah, correct, okay,.

I

Okay,.

C

Hook.

N

It on yeah.

I

As for the owner ref issue, yeah every in the first slow, it seems like if the drivers creating it in manages its lifetime and if it's created externally by the user, it shouldn't be bounded by the drivers lifetime anyway. So.

L

Yeah.

I

By right, except it just means that someone add the owner ref off the driver pod when it's created.

P

Got it yeah? That's your call. We have to solve those issues like that.

P

Besides pot, yes, I'm going to push out on my last unit tests, I've finished and then we should go for full on that review. Otherwise is fun. The last thing is that I think over data graphically Jenkins is hanging. Right now should probably go study.

D

So that we.

J

Have started thinking about the design of this renew.

H

Start is.

J

Path and it possible also post type, the code for the Reno service path.

P

This would be phase for practice. Yeah yeah, yep.

I

So would it be possible for you to give like an end-user demo type of thing next week of this whole process like there are integration, tests and everything, but to see a KDC, and you know creating a token and a couple of flows might be useful to understand.

P

Sure I can do that. Yeah awesome.

C

I'm here to minus 20, so I don't try to do, but I want to make sure that air flow gets us onto.

D

If their.

C

Credit, yes,.

D

Yes, I'm sure I would absolutely look like and I go into premier fussing.

D

Yeah, so there's been a lot going on in the air, take one ah start being a lot of bloopers on my I, my microphone, but uh yeah. So I never see Cooper News team last week with Anirudh, as well as a few members of his team, and got some really good feedback on that. uh We have grant Nicholas and Victor Montero who are joining us and what I would really love to do is I'd love to just kind of go over the basic road map.

D

I don't set up on you reading my discuss to just have get a sense of like what you guys think would be the best routes forward, how to make sure we're staying inside of like today's best practice it and AH yeah. Let me let me quickly: grab both items and I'll, just gonna dump them in and whatever we can get through in 20 minutes. So.

D

The first thing that we wanted to discuss was uh using custom resources for deployment. um Can someone turn off their might their microphone I think that I'm just getting a lot of feedback.

D

Awesome, yes much better! Let me see where we are.

D

What only reason I had discussed was that, by using custom resources for the deployment, we get a lot more flexibility in what the users are capable of doing, while also being allowed to use like highly opinionated stances in what we allow the airflow users to modify versus what we think are things that should be very static.

D

It'll also make it easier in terms of like being able to attach secrets and amount volumes, and a lot of that logic can kind of be abstracted away from the airflow internal code into these custom resources.

D

One downside is that the current release of open ship does not allow for these custom resources, but, after speaking with Eric last week, it seems that in the next release of OpenShift, which should be in the next couple months, there will be they will be using kubernetes 1.7 and then that that will fix that problem.

D

So I think that, if there's almost as though they're like any strong objections, I think that uh being able to work towards like the future, as opposed to kind of keeping things in the air plus source code, where it would become problematic for maintenance for three months for open ships, I think would be a fair trade-off.

I

Yeah I just like to clarify your when you say see: Arnie's you mean in a CRT and custom controller package right where gathers some controller, reading a CRT and reacting to it, but all of this code lives outside of airflow, so it has more flexibility. Yeah. Thank you, yeah. This.

C

Is probably these like not the time to do it, but I'm kind of interested in learning more about the details, mechanics of how you actually use CR DS to do the thing you're talking about because I just want learn more about it.

D

Yeah I mean we I would definitely be find. I talked about that. Maybe we could chat up up the meeting, but uh I know that for a lot of the work, we're doing because we're kind of trying to abstract things away from our users so for one example is like if we want to launch a spark cluster rather than forcing people who might not necessarily know those details to like say: oh, we want four gigs for executors and sixteen executors.

D

We literally just give them the option of small, medium or large, and we kind of take those strong stances, and in doing so we simplify it for the user, I.

D

Think that this would be especially valuable to the airflo community and I. Don't think that the average airflow user is going to be like a kubernetes admin. Who knows a lot of the internal details of what's going on in kubernetes.

D

One other thing that another thing that seemed to be discussed was basically that trying to merge in the operator, and the executor makes for a very large clunky PR, and so it might be better to kind of merge in the just the kubernetes operator code as the first release and then kind of build up some goodwill with the airflo community as well as get in some of these. The changes we want to make to the airflow internal code. Before we try to merge in the executor, um grant and Victor.

D

Do you guys have anything you want to mention about either the things I talked about so far.

J

Not really, okay,.

D

No worries yeah.

C

I'm sure executor, what kind of exec you are you talking about so.

D

Airflow air flow has was what are called it, so air flow operators are kind of what's on the user side, so the user actually defines like I want this to be a Python operator, and then you, the Python function or I, want to be a spark submit operator. You put a spark to make commits. The executor is actually what works internally. So what happens in air flow is that air flow has an internal scheduler and every time that scheduler is a heartbeat.

D

It checks to see if there's any tasks in a queue, and if there are any tasks that all their dependencies have been met, they then use the whatever executor has been defined to say: hey executor, we have this task. You figure out a way to run this task so right now, air flow has like a celery executor. That'll use pickles to send it to a celery slave. It has a Mesa executor that also uses typical, listen to a Mesa slave, and so with the kubernetes executors.

D

We are taking a slightly different approach where, rather than having these static workers, we would actually have it that each individual task would launch a brand new pod to give significantly more significantly more throughput, as well as a makes for significant thanks for significant benefits in terms of like not wasting resources and not being a bottleneck.

D

So.

I

I'm hoping what quick question on this toward some pepper data and found here, what are you guys use for like workflows in to visit traditional luzie type of ropes? Those are you use their flow. We.

A

So we're not direct users right, we're indirect for our customers like.

B

We.

A

See a lot of oozy and we see a little bit of airflow um I, don't think that anything to take away from the airflow. It's just you know. We have a lot of customers who have things that are in production for a long time. So they've got a lot of history. Okay,.

D

Yeah, so actually one thing I did want to really bring up with uh with the general kind of larger consortium, because when when Anirudh and I were talking on Thursday, we, we weren't really able to figure out like a really good solution. For that this and like I guess the the.

D

What we're kind of trying to figure out is a best solution for data for the actual transportation of the dyed files, from the the kind of main airflow instance to the slave instances, and the problem we ran into is, according to the members of the kubernetes team, the git repo volume actually was not made with maintainability in mind. So there's a lot of kind of code concerns with it.

D

So like originally, the thought had been. Oh, if it's like a smaller, if your dad files aren't that massive, you could just pull it from github. Otherwise, if it's super large, you can use them sort of like a distributed file system.

D

But since now, I know that the git repo volume is kind of questionable, like one thought would be actually placing that placing those kind of placing a special case for github. In the custom controller that we're building but I'd also be open to hearing, if you guys have good experiences, any form of distributed file system that you would recommend it's kind of like a baseline for transporting these tags between the master allayed. When.

C

You, when you say a large or a large order when.

D

I spoke to the Airbnb airflow team.

D

They said that Airbnb at least, can have like dag bag directories with like thousands of dag, and each of those bags can be like, like tens of thousands of lines long, but that's, of course, that's of course, kind of the extreme example, and one thing that I think might help that to some extent is the fact that, with kubernetes, we plan on doing it on a like, with the kubernetes executors, we're taking the strong assumption that people are going to be only executing only like people are going to be owning a single airflow instance per namespace, which also would make a lot of sense from a security perspective.

D

Yeah.

M

um

C

This is what I'll say: we're one spaghetti against the wall, but there is like that plugin for get get large file, support, get.

D

Large file for I've heard that what is what is this.

C

Yeah, um like I'm, not sure if it's like actually github specific I'd, have to look into that, but I found it I on it through some github blog, where you can basically install a plugin, and it's specifically designed to support like managing very large files through your git repository interesting.

B

So.

J

Does the kit reporting up here as I saw system volume or something or equivalent.

D

So I, don't even think you I think would get repo. You I'd only think you need to make an actual persistent volume claim. You can literally just mount git repo as like a persistent volume and then what ends up happening is that it pulls from github. Okay,.

J

A.

D

Kind of soft gap is that we could basically just say if you chose get mode we'll just add in a like pull from like pull from github command to the container that you're launching.

A

Are are you worried about the I mean you sound rather large? Are you worried about having every container every pot simultaneously pulling them from github, or is that not a concern um I.

D

Don't think that's a huge concern, because unless you know, unless you're I'll say that if you're pulling so many Pollock parallelism is a setting in airflow that you can say this is the max number I actually want to run at one time, and if you're running that many tasks, then that's kind of like a that's. But that's something that you need to manage incentive like if you're running ten thousand tasks, all of which pulling 20 megabytes each, then maybe you should just lower the parallelism level or the other thing that we did.

D

Is that we're basically opening it up as a layer file as this concept of air flow has this concept of plugins, where you can build custom, executors, custom operators and stuff like that and air flow will treat them as native we're, building like a custom.

D

Basically, dagon quarter like a dag, important and basing we're going to offer github and maybe two or three others, and if you really, if you, if you have like a large scale solution and you have some preferred distributed file system, we give you the ability to just basically create your own file amounts and then that will immediately attach you to all the slaves. So.

A

If I understand that right, you're, basically saying in a normal case, you don't mind that there's all these multiple requests to an external resource that are going to get the same answer because in most cases you're assuming it's not going to be so big that you need to make the network local the cluster. Exactly.

D

In.

A

The case where somebody has a really big one, you have another option of a way that they could use, and so then they could get their locality exactly.

D

So I mean I realize that the distributive there's no there's no good answers in this. The distributed file system world right now, I know that, like NFS has a lot of issues with like locking and stuff like that. But there's there's been some stuff. What anirudh? What's called Mineo or yeah yeah, so I mean like that might be saying like something that might be worth looking into in terms of like I. Don't know if anyone's working on this Anirudh but I I might be if it was working.

D

I might be interested in making a mini Oh like persistent volume, just to kind of offer it up to the airflo community. Saying like hey here's, something that we've at least had some moderate success with.

J

There are some very oh yes masculine with it. It's mean you're, like class 3 yeah, oh yeah,.

P

You can grab your zoom.

J

Kermit, ok, there's.

C

Been some problems with Niall like we were exploring using it, Burano I guess, look kind of similar use cases like you know: containerized s3, inside of a coop, slash, openshift, cluster, okay and I'm, not sure like to what degree. This was like some weird intersection of men I own spark, but there is some bug: that's preventing us from using it, and it's been outstanding for a depressingly long time now and I.

D

Know so the men I owe like community hasn't been very responsive to your issues. Well,.

C

Yeah I mean they were responsive in a sense that we got like you know some diagnosis and some roadmap, but it had to do with like upgrading to Hadoop like 2.8, which of course you know itself took a while, and then they claimed that he had to run some kind of cross-platform unique test found some reason they didn't like. For some reason it took forever to like run this testing I. Can it's not everything I mean currently, people use Mineo, so I'm like it may yet, where you're useless, okay I mean.

D

The the the two thoughts I'm having there is it. On the one hand, it concerns me that it takes a community that long to handle something with such a valuable use case, but it's also that I imagine that, from what we're doing is we're a much simpler use case. We would literally just be like running a simple like hey give me this folder or whatever, but sorry I cut them out there I'm.

J

Surprised I was just going to say: I'm surprised, to hear I mean I would depend on us to version, maybe maybe indirectly not directly I. Don't know yeah wait, Eric does manao depend.

D

On the dupe.

E

Well, I.

C

Think I don't know if it work has a hard dep on Hadoop I mean it may again just be forgotten, like all the details, but.

C

If.

J

You want.

C

Rock depends on how to amend.

J

Anything that the content, library, that Spock is using should be fixed or something like that. I.

C

Don't think it was this was inspired. This was specifically some kind of like they were breaking some s3 contract and I.

A

Think it was like it was on the writing.

C

Side more than my peers, but I think it sort of I purely read. Only it may not be a problem, but like spark want to have might access to I will find it? I will find this thing. You can take a look at it and see if it applies to you. Yeah.

D

Absolutely I mean does any there's anyone else so have any kind of a potential solutions or suggestions for large-scale file, storage and transfer, and especially, if there's any way that we get kind of like minimize minimize the transference as much as possible, because what's interesting here, is that the reason we need to like ship the whole dag folder at once is because there might be interconnectivity between like different dad files, but if we're setting it up in a way that like but it might, it might still be.

D

That person only needs this, like one file or like two of the files, let's to connect to each other. It doesn't actually need to pull in the entire directory. Can you serve? Don't interfere like an.

M

Http or o, or it.

D

In a sense of like a web dev like.

C

You know give it like you: can your stuff read the directory off of you know just over like HTTP URI.

D

I mean we looked, we did take some time to look into WebDAV solutions, although I think that um I'm not I'm, not sure that that would solve the problem, because people would still be like importing the entire directory, which would basically all the same things. What the good thing is, I guess.

C

What I was what I was wondering was like if you could like automate it do kind of like a source to image kind of thing where you know they give you the thing: that's off github and then it you know, generates a container that has all that mounted. You know no.

D

Ok, I see what you're saying like in a sense of like have have some sort of a web dab. That's just continually synced with github and in doing so like the slave instances, would query the web dab instead of the external github, which could kind of reduce the possibility of like overloading the github instances, and also we could just have like multiple instances of this web dev running to kind of keep up with the scalability of the system. Yeah.

C

And I mean if you set up standard like s2, I type, hooks right, you could even you know, so somebody pushes something new up to github it'll automatically. Sync through the hooks known. Could you see what that's.

D

I sourced.

C

An image: okay, in your case, it's in a sense closer to like data to image, but it's the same idea: okay,.

D

um Yes, I mean that's, definitely a possibility I'm, just okay, so.

J

Everything is balanced, I mean similar to what we do with the research staging server. Although.

F

Is not.

J

Really that bad, so maybe.

C

There's a chance you.

J

Guys could take a look at this research staging server and then maybe refectory yeah.

D

Yeah I would have absolutely appreciate that. Is there any chance you could message me on the kubernetes slack and just give me the link to so I could just look through that code. Man.

J

Will be the live person actually to give you more context about this yeah.

K

It's a yes, it is just as simple as just a simple HTTP server. What visit, upload and download endpoint there's always.

B

There's.

K

Probably someone from use you can make to it with a to make it scale out better, because right now, just sorts on it just sort of darker disk. It doesn't use any and although I guess you could configure the volume so pretty pretty easily yeah good one optimist and and and I suppose it uploaded a download end point we're just like just takes an input just like you just provide it with the cursor diamond data input and it looks like a treatment in on output. um What's what's particularly unique about the resource ad server?

K

Is that once you push data, you receive a secure token, you have to provide a secure torque in order to receive the file, so that basically gives you like. It, gives you a very simple and yet for our use cases it's mark at least effective mode of security and authorization where you have to like the only thing that uploaded this thing can be the thing that downloads the same uh so I don't matter, but for that I'm not too sure.

K

If that would work for day or full use case, no, it seems like something uploads the thing, but multiple people want it down. Multiple instances want to download the same thing. Yeah.

D

I mean.

K

Ps not maybe as a find a way to share that token across multiple, although things that want to download it. But that is nothing answer so.

O

I think Jude's kind of like- and this is grant by the way, um there's like two separate problems here, which is one getting airflow to recognize the bag and then two getting all the workers to actually like pick up the code, so they can execute it and I think those should be separate because publishing like a dag, so it can be run versus you know actually running.

O

It are two separate things, and so, whatever like, what I wouldn't want to see would be like when having a solution, we're uploading code to some like you know some some web dab type thing where requires, if you have like a dag, that's already running, would like over write the code, that's already existing, so just kind of like separating up those two things. You need to be careful.

O

So the example would be like if you, if you publish one version of code, you had a dag that was running and then you republished more code, and then it would clobber that existing code and it would break this.

D

Isn't so wait you're talking you're talking in terms of like it's like halfway through running a dag someone changes the bag file? So um yes, I. Definitely agree with that.

D

I think that, and that's one thing that was kind of given to us for free with the with the git repo vol mountains that gate repo volume mount allows you to specify a release, a specifier of revision, so that you could have a guarantee that, like as like, as long as like there's like some mapping of the dag the bag to the revision, that's being called every task will always run exactly the same.

D

um There is a possibility we could set something up in the HTTP server such that it kind of just keeps some mapping of revision to what like what is serving.

D

What grant? What do you think about that, like just like kinda having like that when you do the query, you have to query it with the reverse revision version and when the executor creates the creates the slave instance, it creates it with like an environment variable or a config map, just pointing to the like with that revision, so that when it starts up and knows what to pull yeah.

O

I think that's really makes them cool um bill. The one extra bit would be like having everything be at volume is like. Okay would be a little bit nicer if you could choose kind of any persistent volume backing, but kubernetes has a nice abstraction but I guess it's not super important. Well,.

D

So I mean the the problem with the other backing. Is that it's the same issue, which is just that less.

B

You have.

D

Some sort of guarantee of revision: we can't guarantee that the latencies are getting the right version. Sorry.

I

To interrupt this is a great discussion, but I think we're running a little bit over time. I'm getting kicked out of the conference room so now yours just gave you a bubble eating where we discuss these things in detail. Yeah.

D

That sounds great to me. Sorry,.

I

To cut you off again, no.

D

No worries I'll message: you guys on the clock and try to get a kind of clear. We can coordinate a little brother. Okay, all right, awesome, hey guys, Thank.

O

You nips thanks. Thank you all.

D

Right.
youtube image
From YouTube: Kubernetes SIG Big Data Meeting (August 16 2017)

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).