Kubernetes SIG Network, 1 Jun 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Network 20170601

Description

Kubernetes SIG Network meeting 2017-06-01

A

In recording is on.

B

Cool first thing on the agenda is the 21 issues in the 1.7 milestone.

B

Down setting you added this, we hopefully already saw we we try to do a little bit of curating of open issues and bugs that we think we should make an effort to solve in the 1.7 milestone.

B

C

B

House, where.

D

There's an engineer that.

C

Are relatively easy, although I think we've had enough new people who popped up to say I'd like to take bugs that, maybe these ones are all taken. There are a few in there that are going from small engines for sure all, but I don't want to discourage anybody. Try to take those on just a little bit more application, fresher and honestly like. If you are one of those people who wants to contribute, but all of the obvious bugs are taken.

C

You have to only work on things on that list, but we like sit on PRS until after the merge window.

B

So for this meeting, what what's really our goal be with this list? Should we be identifying the top priority issues that not.

A

B

On yeah, do we want to present it and run down them.

C

Terms resented on zoom' before you. Let me do it described right. Let me pull it up. Why don't you hit the next topic while I pull it out, yep.

B

So next topic is open flakes, so there has been some activity on flicks I've, seen a few new ones popped up and I.

B

Send out a list.

D

B

Folks, has anybody had a chance to look into that yet I know a few people have? Is anybody looking for help with a flake that they're currently owning.

A

I'm working on four six, six, five one, which is the latest one there since I, apparently caused that one. So.

C

Yeah yeah, then I got the an earful on that one. Yesterday, sorry about that I'm.

A

Still not entirely sure why it's happening that way, but I'll get to the bottom of it. I guess one question I had with that was: why does pod status get skipped or not updated? If the network plug-in returns an error for the sandbox, we shouldn't an error for networking, cause the pod to be torn down and restarted, or something like that I mean it seems like. We have ways to report. Network errors, no, but they're not actually respected, and nothing is done with them really.

C

Sorry then um we we cut out or you cut out for the majority of what you designed sorry.

A

um I was wondering if it seems like we have ways to report Network errors now, at least through the you know, get pod network status should that be respected, because I think right now, cubelet doesn't do anything with those it basically just logs them, and then he like doesn't update the pod state at all. So they basically get logged and lost and nothing gets done because it comes.

E

E

I feel like on there. The implicit requirement for the sea is fact that the get hot Samba Paris she basically in return error where the limit has followed on. So that's why Mike Kidman right now, it's pretty like, like it, has a dumb book just to go through all the parts and then get to each one of those seven dog status and then, if any error occur for that particular part and that hostiles will not be updated and then you will get stale in the API server. So that's sort of the wider.

E

The I effective and then like I, get the problem but yeah, but I'm not sure like how we need to attack this because I think the ultimate problem on install is that like during, like hard start up and tear down pod status should still be valid right and it should not meet any useless error right. That's the the problem. We wanted to get rid of.

A

F

I mean I, guess.

A

I'm so I mean if there is a network error, some kind of well, the pot is alive. What actually acts on that and then tears down and restarts the pod. If anything, does it just get left to health checking.

C

Can you be more specific what sort of network area you're thinking about.

A

Well, I mean in this case there is no network namespace for the pod, because key book got restarted and for whatever reason, doctor is not giving us a network namespace and so clearly that pod is not going to work and the error was getting reported. But then that causes the status updates to not happen.

E

Yeah I mean yeah, you can you can say that from the other angle, where your your icon, cubelets should probably like act on those errors, um but it's it's hard to formula to tell like. Is it an intrinsic error or is it like a real problem or basically it's very hard for two with the decide? What is this error section.

E

Again, we are singing like you're saying that, basically, if okay, the status, guess that is called returns, an error and then as Cuba should cut back on it and they should basically tear down and restart it. And things like that. But it's hard for people to tell lies a specific area of return and then based on that to to to like to act. Should.

C

We should we be formalizing the errors there like. If there's actual errors, perhaps we should be returning something that we can detect. You know is actionable so yeah.

A

D

A

That's the specific issue in my second question, though, is what, at this point, is basically nothing pay attention to network health, mm-hmm, you're, right, you're, right, okay,.

C

They're right absolutely I think there's just an assumption that, once the network is up and running that it's not going to permanently go away. Yeah.

A

I mean I, guess I would that that seems: okay well cubelets running, but if cubelet stops restarted, then kind of all bets are off there.

A

Apparently, in this case, the only way we have to figure out that something's not working is that we don't have a network name space, then the plug-in can use, but so um well I'll try to go figure that out information now I'll try to see what I can do about errors too. Turning from get pod network status, I mean the core of that original PR was just to try to make sure that we didn't log superfluous log messages due to the race with the pod lifecycle generation.

A

So I think we can probably get that goal and not quite get to the point of figuring out network errors. Yet.

E

A

Okay, that's it.

C

Okay, well, we have other major flakes that we need to get an egg out of this month. I mean okay, let's really.

B

I'm not aware of any at the moment that our release blockers but I've just added a couple new ones to the list. So they haven't had a chance to dig into.

B

It does anybody have like that they're investigating that they think is release blocking I.

C

Want to encourage people to begin to flakes too. Even if you don't know the subsystem well sometimes staring. If the test will show you what the flake is sometimes adding more logs is all you can do to resolve the flake, but you're getting a mr. small is better than letting you say it.

F

C

Family I'm trying to present my screen. Can anybody see it yep? You can oh great I'm, not okay, so.

C

All right, we have 21 open issues, one is a flake. Let's just run down them, I guess, and what do I do just like touch on what they are or status? You can be owns them if anybody feels like they own any of these issues. Please scream at this point.

C

The person here is IP mask agent. First, two actually docks and setting resources generally. What is the F domestic agent? ah Sorry, so cool, so IG mask agents. So cubelet has this flag for non masquerade, cider, which is needed on some platforms like clouds to enable egress, for example, GCE. Won't.

C

Let you pass the edge map if you're not coming from a VM, so we have to masquerade if you're trying to go out to the world, but you own one at mass great, if you're going out to the world, so we had a flag in cubelet. That said, don't masquerade for this particular cider and we wanted to expand that to all of like all of RFC 1918 space, but also to make it more user configurable and rather than add, more Network flags to keep what we thought.

C

We would instead make it a demon set, so they were Daniel and Mike tough and wrote this little IP mask agent, which takes some configuration from a config map and writes up the egg potatoes rules to turn on or off masquerade. As me in and the nice part now is, we can remove one more flag from cubelet in every search, so so there's that it's new we're trying it out on GCE. We expect that it will be a good thing and that we will recommend it for everybody who needs this sort of use case.

C

So Daniel I, guess you'll, almost two one ready to merge. Ok, we sort of afford it cool all right. The Daniels on those two nice tables procs here doesn't respect, find a dress that one.

D

I should say are open to that. Yeah yep.

C

Yep cool hairpin not set with cubelet with C and I plug in pink spec. Do you yeah.

B

Yep I think that.

C

B

Think the latest on that was that it was just a documentation thing for toes a change that wasn't editor release, notes and I. Think we can close it and then opener Doc's issue to write like expected, behavior for plugins, or something like that.

C

Okay, you know that one or does somebody want to be responsible for that. One.

B

Nobody types up, I can take it. Okay,.

C

Last chance all right, it's KC's, cubelet iptables performance issue with cubelet in 19, April stuff. Now, there's two issues here: there's one cubelet is doing more I see table belts than it really needs to. They could be doing it through a saver store and two at least on some machines like looking cables, is 10x slower than it is on other machines.

C

Are we doing anything with this.

G

But there's a guy on that. Our thesis got ourselves taken up like a large, so I'm working on it for a while, but I'm, not sure if it's made any progress. Okay,.

C

I've had a chance to review email all day. Oh.

G

This is where you I have seen linked issues about certain kernel versions, but I'm not sure if that was that's really related to the original issue. If it's just an issue in general.

C

Yeah so min Han, you identified a particular kernel version. Deep. Do you believe his kernel actually does not have the performance regression? That's based on the color.

E

Version does not have couldn't regression play.

C

E

um The badges look I'm not sure, particularly what I don't know like that's. The specific goal is probably the the patch did not come rather I'm, not sure, okay, but it does look like exactly like the Cobo. It sounds.

C

Like it is good.

E

C

Seconds saying on IV needles.

E

Okay and then, and then the upstream folks, they said, say: I'm not sure whether they have covered IP tables, that Sea Kings and.

C

Probably I seen that's exactly what we're doing. Yeah that's more than I, see: okay, okay! Well, that's interesting!

C

So this bug is nobody as far as I can tell nobody's working on the fix to move it to IP tables spades and restore new non. Can you take this issue, but try to get this guy? Is this our theist guy to drive the patch sure the hills new contributor out of this okay? Okay, I? Would love to see this fix, but I wouldn't call this p0 like I, wouldn't stop the release for this one, because it's not a change in one fix. So this is probably not a blocker objections to that that.

B

Sounds fair to me: okay,.

C

Service controller update load, balancer host drains all external announcers during full master disruption. Well, I do not have contacts in this one.

G

You have a comment at the bottom like six days ago, yeah.

C

I read all of these over but I, don't remember what they all were. Oh.

C

Right this is like the very small cluster where what why were they checking out I.

B

Think they had a book in the Senate in the API or something and then those ah get active. The.

C

Clothes weight issues all the Masters that were stuck with all their sockets and clothes weight, and so it couldn't answer any it couldn't take any more connections, so all the cubelets timed out, so it removed them all from the load balancing pool, even though the cubelets were actually there and surveying they just couldn't check in yes, yeah hmm so like in board. We have a system that that recognizes too much correlation and puts the brakes on pair actions like this. We do not. We don't have that income and yet.

G

We have like an exponential back-off mechanism, not enough that could be applied here. It's sometimes used in some components like the couplet, where you're just retrying a bunch of things, and that puts a lot of load on the API server and then things get out of whack.

C

Yeah I think that the problem here is that the clusters were small enough that, like that, wouldn't even really kick in.

C

Like if you have a three node cluster and all screening of are believed to check out at the same time, like that's one thing but like if you have a to node cluster and like, is it impossible that you could check out? At the same time, no back off is going to help you there yeah, um okay, anybody has bandwidth to look into this. So just even ID a a on a heuristic here, I know the controller logic is probably new to a lot of people.

G

You can assign that to me I, don't since coach Frese today, I think about it be removed from the approved for this milestone, but.

C

Well, yeah, maybe we could. We hardly have to make an argument that this is a bug fix and it is actually sort of a stretch, because this is not a new. This is not a new in one seven ish ooh I'll assign it to you pontificate on whether we can possibly spin this I'm, not sure that we can. At this point.

C

um Where were you pods with host Network feels true, cannot be removed and generate errors, we're using C and I.

C

Cni experts I just.

A

Looked over this one, if you go all the way to the bottom, I think four, three, eight seven, nine we'll fix this because I made the CNI tear down, not depend on namespace but of course, for three eight seven nine got reverted and six set up tapped again: okay,.

C

So can we stick you with this one? So whenever you read re River yep that can this one yep.

A

And then also, let's see when you say, stick me with it: does that be in the setting, yeah I guess so. Look I'm on planning, because my windows are almost four to seven. Thirty-Five is also would be salts by that as well for what it's worth.

C

Which one is that we're just a little later.

A

In the list that is cubelet, C and I, innocent or failure.

A

Awesome come on github I, just defend myself! Well, perfect.

C

Okay, cloud provider, public IP resource leak, Linda, leading and hitting local services. At the same time, yeah I've read this one. This is a good one.

C

I'm not sure if it's Audrey, specific or if it was applied to all Club writers, I have not audited the GCE code, but basically, if you create and destroy a service with the same name too quickly, you can overwriting your state and lose track of the fact that you didn't complete the deletion of an IP address. It's.

F

Because the name is not special, when I do ID.

B

F

Go with the conclusion of it the bottom eleven. This is the resource name in better. The form is like a da Lang or the things its main thing, but Aldebaran.

C

Yeah and we still no matter, we have this problem. If you were in the middle of deleting a multi-course thing, yeah.

B

C

Crashing come back, you know ideally yeah. So that's a big change. Yeah.

H

C

Not a bug fix that we can tackle in this boy I'm going to sign to see you. Can you update it so I can't.

F

You mean before balloon: yeah.

C

Or did you know, comment on it and say that.

C

C

A flake in the center Keeble it takes a long time to update pot of pee.

B

Yeah, that's me, so that's that's something I'm still investigating, but.

B

Basically, I'm seeing it sometimes take upwards of 30 seconds before from the time that it, the cni plugin execution, is complete and the qiblah tries to report status to the point that it actually shows up in the api. Isn't.

A

That expected to look so that some of the cublas status loops with 30 seconds I.

B

Think it has a kind of periodic resync, but from my read of the status manager code, at least this, and in a lot of cases, it is occurring in less than a second okay.

A

There might be some.

B

A

Node status in the pod status, yeah.

C

A

B

C

Why is it sometimes missing the Train.

B

Yeah I'm still trying to figure that out, it seems to occur when I start a bunch of pods at once, which makes me think it's.

D

B

To the kind of queueing of of updates there and perhaps so, sort of rate-limiting in the API, but I don't quite have a grasp on it, yet I'm, actually digging into that today. Okay,.

C

Cool we'll leave this one open for now my great bells and whistles to see and I. This was this is VM I, believe it was a traffic shaping thing and passports. So host ports is what is the status on my host port, seeing that.

B

Merge merge Nova.

C

B

A

We need to reinvent.

A

Ursula and I yes, he's.

C

Been doing that mostly, they even help you, okay and we're not robbers you're, not doing traffic shaping sigh hell, yeah.

A

Diane occur, which I'm fine with I. Don't know. A lot of people were using it. So once we get the updated CI binaries and we need to update at least cube net to actually use the chain instead of just a straight-up config. So.

C

We should think about what I want to make that change at this point in the recycle or whether it's in today, if you use seeing hype where you use yourself, I would say no make sense like I'm, not sure the women team know that does not make sense or no. We should not change it. No.

A

I was saying: I, don't think we should try to get this four one. Seven, okay, two.

D

Key means out Morgan wanted to is that being impossible, because the filters to compile your product and I just think really? This is a CNI issue. We have to go back and figure out. What's the plan for cleanup okay.

A

Quick touch that it is pretty quiet. Could you repeat.

D

So I guess there's somebody not. We use the pond IP address' right when you're cleaning it up right. If you have multiple traffic shapers when you're cleaning it up, you don't know your pot IP.

H

D

To do is the fact see we do that. We're both hash the pot item up, but that's not really a great way to do it. I.

C

Mean this goes down the.

C

Checkpoints that diminish the person recording you guess we just need to come up with an answer best practices, but it doesn't matter because this think the traffic shaping not going to be in one seven. There's no aspiration to that. So this bug is currently a certain time to been Chris and then do we want to just update it with the fact that the course port stuff is merged, and we can remember the plot that my area so that somebody law, so you did in a custom, CNI a config, yeah and so.

F

B

We can, in.

C

Computer education support sports now, even though it will work.

C

It'll still working.

C

E

I think this is fixed, I think the there's a there wasn't change that affords it to update and that's it, but the the co has changed so much here. You can okay, we check your.

F

Check out yes, I.

E

Can all right I.

C

Love this, we don't have to protect me if they're just already done another flake.

C

Raised between Q proxy start up in node registration. This is a good one.

E

So, basically, is this politically.

E

But here's the thing so cubelets pops up and then it needs to register yourself and then meanwhile, there's a bunch of static pods. Where.

E

It starts to sink those static bars, so these two things happen in parallel and then does some static power like cue proxy, is like something needs some information from the nose back where, depending on the note getting itself registered so I guess so after we move the cube proxy to the you said, this will be a problem anymore, but at this was still a problem. Is.

C

It is cue proxy I think I should know. That's very dumb is cue proxy, literally looking at the note object, or is it using API to excessively I think node IP is available to John when API you know calling client chord notes.

A

Get hostname, so it's literally going out reaching out to the node and seeing yes also note that whatever fix is here for this issue is going to conflict with the fix for the bind address there. They touch exactly the same code, sort.

C

Of well, you can conflict from our code level. Point of view not from our concept from you, yeah.

A

C

This seems like a rel and like can we just do something dumb like hey the node IP is empty I'm, going to admit and then like when, eventually the back off will start, and it will be a backup for resolving your node IP will be there yeah. Could it be something dumb like yeah right.

E

C

E

Get a total side yeah. That sounds like a terrible idea.

C

Okay, okay, I mean I, can't I can't think of a reason why that's really useful I mean if somebody wanted one cube proxy. You stand alone on maybe I server, I guess. But what not is not.

E

The guy, so we know.

A

E

Arias way, oh.

A

If you want to run it standalone, then defined address PRS that gets committed. Then you can just tell it what address divine.

C

That's true, yeah um yeah I would say if we have to get the node IP and it doesn't exist that we should just either exit or like sleep for ten seconds and try again and put that into a back of me. Yeah.

A

I would say if you don't specify a bind address, just sleep if you do specify, bind address and I basically use that and don't hit up the API server. That's.

C

It so I mean probably it to be a little bit more subtle than that. Like try five times, you have a hold second delay in between them and if it can kick resulted after a net of thirty seconds, then it well.

A

Like exponential back-off, something I think he said: hey yeah.

C

Sometimes we have a utility- oh just like something that seems supposed you. It shouldn't be more than four or five lines of code fix this all right, I've assigned been on like five bucks already, okay, somebody else want to take this one community, people, volunteers,.

C

I

Going to I'll take that one who's.

C

That one Mike yes, a mighty, carry you okay,.

I

I hope we'll see actually five stupid question on the code base. What do you recommend for me managing dependencies? It's not a.

C

So I think glide was a cool tool, but it didn't work for the kubernetes repo because it was too big and it had to neared interdependencies. I'd already, remember exactly why, but it fell over. We tried to run glide against today, so right now, the only tool that we found that works at the size of our repo is go down and go deficit turn.

I

Yeah I heard someone else tell me on one of the current slot accounts a couple days ago. uh All four new projects: okay, yeah yeah,.

C

Like if a new project you to start from the beginning, I think glide is a better tool until the real go depth tools available. It.

A

Is a question here Mike how to just search bootstrapping with development and get it built enough to be able to test it? I'm.

I

Not sure I have an actual question. We just get into it and see if I have one I'll call out and slack if I need help. Yeah.

A

That's fine I mean I'm happy to help get that bootstrap stuff up and running. It's pretty easy to just build the parts that you care about and to do something like a local up cluster. To do your testing.

C

Because notice, Mike you're not marked as an ordered member or external collaborators, I can't assign you the issue so I just tagged, you I think I spelled your name right and I will take a note for myself to go and add you as an external collaborator, so you can actually get assigned issues and PRS. This is true for anybody here who, if you're, participating, you're working on issues, and you want to be able to be assigned an issue or a PR drop me. A note and I will add you, as an external collaborator, remind.

I

Me what this means I mean I, went through this corporate CLA thing years ago.

C

What else we're talking about now? It means all it literally. All it means is that you're able to be assigned issues and PRS okay, what when I go to put in your name, they will automatically complete it and it will be allowed in that field. That's the only power it comes with. Okay, thank you.

C

Service can change from external name to cluster IP I. Think there's a PR open for this. That I have to review.

C

Validation, enhancement, blood valves, referrals, no link what's on means.

F

C

It's a la animal.

C

And it sounds like a bug report. The.

C

Port is not working, so it looks like okay.

C

Somebody should try to this is an old one. That to me needs to just try to reap read this. Hopefully it isn't real anymore doesn't seem to make sense to me.

C

Volunteers, like first step just try to repro on something close to a head.

C

E

C

Did who's that up to think on? Your name is Mel your github Braa.

C

Cole same with you, I will add you as a collab thanks.

C

Hopefully it's a I tried to reproduce, and then it didn't reproduce and then you can just close it.

C

End points controller demands, ports for headless services, I think there was a PR open for this one too.

C

Is there appear well no, but chunji one, two three, so they would like to work on this. A share.

C

Any regrets here we can revisit this one and if there's no progress by this time next cycle next meeting, then we will assign it to somebody. Hopefully this is not a hard one. A lot of these are little validation. Issues, expand docs on DNS policy, so a change went in last cycle. I think was it this cycle to add a DNS policy of cluster first with host network, which is quite a mouthful, but that's what compatibility gets you? It feels like looking at this understanding it and runs and docks very volatile tone.

C

So, oh great, that's right, he did cool cubelet flag. Cluster 9 doesn't work, it was all calm. This is another one. I looked at this week think this is a non-issue I'm, I, guess I'm on this one, because I haven't assigned myself any okay, um that's it! That's it hey guys! That's it is it did anybody here feel left out like they wanted to work on some things that didn't have an opportunity to cuz. We can go find more.

C

It gives us about 200, open-sink Network, bugs which honestly, like out of 4,500, is not too bad and 4500 is down from 5,000. Just a week ago, there's been a concerted effort to find and close stale bugs which to all those people. If your presence thank you when you've destroyed my inbox, but it's been great to get those bugs clothes.

C

So I just want to say again thanks for everybody, for sticking with us as we sort of retool and figure out what this thing is and how it's going to work so far, I think the response has been great. I've had a bunch of people contact me asking how they could help a bunch of people going through and doing bug. Scrubs we've got more volunteers coming on here. It's fantastic.

C

Any other agenda today guys I, think yeah.

B

D

B

We have, anyway, from the f-86 working group present.

H

This is Daniel that we have other folks here as well. I guess the only question I really have is Tim yes to create what I call the manifesto and I posted that to the group got some feedback, but I was just wondering where we go from here. Do we continue putting leading in on the Google group for feedback? Well.

C

Great question we, so they governance group hasn't formalized where we're going to put charters, but this is sort of the proto chair. I, owe you an apology that I haven't actually reviewed the dock, yet I've got a whole bunch of musics, related PRS, but I haven't had a chance to get to and because I, mostly because I, don't know how they work. So I have to go.

C

Do some creepy ours and your dock got caught up in the same thing, six net, so I'm going to try to get to those early next week before I disappear. So.

A

Gilly comment I had on the manifesto: was there isn't anything about tests or testing in then I think part of the working groups? Responsibility should be writing in test cases for ipv6 as well. Maybe that goes without saying, but I feel like it should be specified. Yeah.

H

It's a good point: I thought I had it in there, but if it's not I'll definitely put it in there, because that is definitely part of the manifesto or needs to be.

C

Okay, so I promise you I will read it in the next five days or so, and.

A

I think I brought this up before maybe you're not Tim, but is anything a PP six death testable within Google infrastructure. All right, not not running like slowly. Okay, so we'd have to do it in like sharor or if keys I forget, which club letters even do ipv6 Amazon.

C

Has some very limited, v6, okay, but yeah we would have to be testing an usher or you have to set up a little test bench of bare metal machines or we'd have to do you liked. Will you be six on overlay like how awful is that going to be no.

A

I mean just for testing, it's probably fine, I mean the overlay is going to reduce performance somewhat, but we don't care about throughput for testing yeah.

C

That might be an option if we could get the tests if we could get like an e de suite put together that installed machines with flannel or leaves and use that to test within six watch. It.

C

I'll leave it to the v6 group to figure out which which passes the the best, but the big advantage of the GCE pass even with overlay, is that you are already paying for it. We are already running it on regular basis and see if.

B

C

Almost ready to pay for that, so it's actually coming out of the right budget really.

H

Deafening from my standpoint, I just need to better understand the the end end tests, so I formulate a kind of a baseline to work from okay.

C

The tests themselves shouldn't be that complicated building up a new suite is a little bit more complicated, but we can certainly when we get to that point. We can put you in touch with the right people here and get a sweet set up so that we can actually run these six tests. We probably won't run them on an every APR, at least initially, but we can get them so they run every day or something. Let me just no.

H

Question to is, if there's any pull requests related to b6 that don't have the area ipv6 label, what's what's the best course of action to ask for that label to be at adjust within the github conversation and maintain whoever's designer that PR should then available. I think.

C

Opt command for slash area. So if you just a flash area space ipv6, which.

H

Item do that for issues but I tried. Maybe it was an I, don't know what it was, but I tried it on out of PR and a work out. Pr. Okay,.

C

I'll um I'll take that back to our country backs folks. I'll, ask me it's supposed to work through PRC little ISM is area / area space I've even put.

H

A v6 yeah and it works so great for four issues. I see.

C

A case here we're going to on a poll a unique, be no.

B

Direct a little bit of a slug ENS, and this in the past we could to a couple times I have that anything.

C

Because this is on a pole that you're on Dania hmm all.

H

Right, well, let me I'll give it another try of an issue then I'll just open it up. Yeah.

C

It's not working ping, B or ping Garret Rodriguez, G rod, cheer on their G Rodriguez. The Reedy I think it's good rod and he's our main contradicts lead there.

C

All right thanks cool all right, so I have a bug in a couple of action items, and we also got some bugs and stuff to work on this the today of the code free. So if there's any other PRS that people wanted me to look at that need to go in before the code. Freeze today is the day. I will be available on hangouts from now until about 6:00 California time and then again later an evening. So if anybody needs anything from these, please hit me on hangouts or slack.

B

Great thanks all right done very.

F

Good thanks, everyone.