Kubernetes Community Office Hours, 28 Jun 2016

Previous Meeting

⏯

youtube image

►

From YouTube: Kubernetes Community Office Hours 20160628 -- Filip Grzadkowski

Description

Office Hours
-----------------
Kubernetes developers hold office hours over VC where we discuss questions raised on slack or by attendees about how Kubernetes works.

In this video of Office Hours Filip Grzadkowski answers --

dbhanushali - What is the federations_domain_map variable ./cluster/saltbase/salt/kube-dns/skydns-rc.yaml.in

gyliu513 -- Is there are anyway to reduce the size of kuberntes binary? I found that the kube-apiserver is 108M

aps - Is it just me or do k8s unit tests take lots of time to run?

shivania - Are any tools that can help me with inspecting the code.

mgoodness - Is the Kubelet’s read-only port (10255) enabled by default?

andrefcruz - Is there a flag for "kubectl get events" to return ALL events instead of the last X?

andyverbunt - I’ve installed kubernetes on top of docker on my osx machine (for development). Everything works great, but probably once every day, the api server is longer reachable out-of-the-blue. How can I diagnose what’s wrong?

jnardiello - I want to be notified when a process crashes and doesn’t exit gracefully (aka downscaling organically)...

crizcraig - Just started getting ```Error syncing pod, skipping: failed to "StartContainer" for "clari-core-core" with ErrImagePull: "Tag latest not found in repository docker.io/clari/clarius_core``` My image pull secret was working previously. Double checked it’s still there and contains valid credentials. Anyone have any idea?

A

Rena good morning, Philip or good afternoon, fellow good.

B

Afternoons are a high, so one of the questions that I found from iluu 5.3 is: is there any way to reduce the size of Cobra? Need? Is binary I found that the cube API server is 108 megabyte just wondering if there are any compiled options can enable me to reduce the size.

B

So that's an excellent question. Honest I'm, not a goal, long expert, but as far as I know, we don't have any options like this, otherwise we would be using them. So I don't think this is the case. I think in general. Go libraries go, binaries are pretty large, so I guess that's it. It's.

A

Something longer term that it may be worth talking to the NGO community or the go team about it. Some sort of further way to reduce binaries, certainly something we could be. Okay,.

B

Though I would be interested like what is the background to this question, is it that the machine is too small like? Why would you want to invest in this like why? Why is it.

A

Smaller always better, but.

C

Yes, but since this require someone, no.

A

I don't think we need to prioritize it by any stretch immediately, but it's an interesting question, especially these workloads get containerized. People want to put more more containers, more smaller object skin. So maybe that's where the um yes.

D

That's a good point. That's a good point! Yeah, okay,.

C

Let me see if we have.

B

Other questions I feel.

A

Like we should be a talk, radio, you know like hey colors, please feel free to dial in live.

B

Yeah I don't think there is that many people line I like it now there is something very concrete about what is the Federation domain map variable in class, so South by South cube dns, kai, dns, RC, Yamin? Okay, let's see what this fine so I will open the sum it up and we'll see. If we can see what this is and I don't have.

E

Cluster, so sometimes some.

F

E

And cube scratchiness RC gum in ok and now what are we looking for Federation domain mutt.

E

Right right right so.

B

The way I understand this is that DNS need to resolve services between in a cloud in the Federated cluster, so where we have number of clusters which are working together under so-called abilities, we need to resolve services between those cluster, and my understanding is that this variable Federation domain map will point the whole point. Proper services to the different clusters, though, would have to walk through word. This is exactly sad and understand it better, but that that would be my guess. Judging from the name, should we go into more detail Sarah, we.

A

Could or if you could even just leave, maybe a breadcrumb? Where would you send the person who asked this question next? So hey, you might look in this section of the code if you're, if you want to see more, it's.

B

A very good question would have to search for good.

A

We'd have to go look at ourselves, I.

B

Don't know I don't know I can I can try to put this right now. Let's.

A

B

Couple minutes see what we find is.

B

That, because I need to ssh to my work session, where I have all the code, but we can certainly do this.

A

This is fun because I'm going to be able to take this, recording and publish it and then also pingback Beach people, whose questions that we're answering saying you didn't come to our office hours, but we thought of you anyway. Yes,.

A

I'm scrolling back up and trying to check see if there are other good questions. Okay,.

E

E

Okay, let's see Federation come change, Coon's age, so.

B

It looks like I was just craving. Maybe I should actually share my screen or something yeah.

A

That'd be great, I have.

B

Never done this in this Zuma's, but let me see: stir screen okay, desktop screen. Oh.

A

Yeah we see you see, do.

B

You see this useless, yes, okay, perfect, so I was just gripping and the code base ball. It seems you're. Basically, two files, one is come on sh and the other is this. Guy deanna sauceda mean, and I will ignore the last one, because it's something we said so I suggest it. Okay come on Sachin. It.

A

Will only go to said if we have to yeah.

B

Let's see what is inside this file and we can have.

E

What was the name again, Federation domain.

F

E

Domain, what is this Federation? A fresh everything.

A

E

Copy paste works always.

B

Relieved, the formulas is use the reg Federation environment is instead. This is to allow users to provide the speaker, values and regular salt mekinese curing whoa okay, so we should now search for basically this viable name house. It's sad.

E

D

Till there I guess: ok.

B

So this lines looks interesting. This is like Federation name, equal ENSO in the name yeah. So this basically means what I just said: that there will be some names assigned to those clusters and we need to resolve them. So we need to understand that. Ok, this is pretty sure whatever is should be resolved actually in this other cluster, but we can go deeper and see. What is this in.

A

So it's the key value map. Oddly enough of what the Federated cluster name is to what DNS zone it is in yep.

B

That's my understanding is injection and give some example here in Federation cluster come on a stage where we export in a zone name as Federation dot example, and it.

A

Also has to map the match, the RFC requirements. Awesome, of course, of course,.

B

Yes, I, don't think it makes sense more to go into details, yeah I'm, not sure, what's the state of the documentation for cluster Federation as it of occurring in future. Ladies as.

F

A

As it's coming out, the first bits of it are coming out an alpha in 1.3 late, we'll be getting some bits of documentation following us, yeah.

B

That's exactly the case, so the general answer to the original original question was that it's a variable that will help dns to resolve service names from different clusters under federated last yeah.

C

Awesome cool that.

A

Was wrong? I learned things every day here it's great.

F

A

So it looks like somebody was having trouble with the NFS example earlier. Have you played with that example? At all? No.

B

I hadn't, we couldn't see what we can find out when was it so that I can see. Actually the the question of the problem.

A

It was what it was. This is going to help because it's time zones 120 p.m. yesterday. This.

E

A

B

Maybe user, maybe I will search for the user.

A

User is SI, je and see providing. I can see this early in the one. Yes s IJ and see just above match. I think.

F

B

This is the one who you're referring to yeah official NFS example to the troubleshooting and unable to connect the NFS server bait on the example. Okay, this is a very general. It.

A

Is it may be too general? Do we want to pick a different one cuz? It looks like he got some answers.

B

Yeah, I think it would make sense because we would require it would require, having a lot more context, to actually understand what was going on. Okay, read what happened but check your blood logs, you, computer, just environmental, so.

A

There was another question that was pretty straightforward: is a mere decades unit tests take a long time to run. It's been more than 30 minutes to finish. Yes,.

B

Unit tests are another, the best shape that they could be, so there are two things which which are slow. First of all, just building the binaries. That takes us a lot of time and 108 days. It's not only because of the size. Actually, it's more because in the default by default, we are building all the binaries for number of platforms. Yeah.

F

B

For article whatever else like number are my thing and then dionis and something else, and this basically linear increases our bills time, and so yes, it takes a lot of time, but then I think that unit test itself should be faster than this.

B

It's more than 20 minutes and tests.

E

Are still running yeah, it took 30 minutes to finish this.

B

Is we have exactly the same problem in in our pre submit checks that we run on github? We are trying to fix this now, but I'm not sure what the current state I know.

A

B

We started dividing like integration tests from uni test to speed this up and stuff like this, and this has improved a lot. Time takes a lot of time so which we should definitely look into this more carefully. So.

A

Definitely something to be worth done. Yes,.

B

Yes, yes, yes, I think that mostly daniel is leading all of this work. I asked me but yeah. We should probably ask him about more details.

A

So I have another one who the 25th that looks um pretty broad. I want to inspect the kubernetes code since I'm planning to implement a custom volume plugin, which is ambitious. um Hence I would like to know what tools there are that can help me with inspecting the code. So maybe a go tools: / / you! This was the morning of 25th. Well in my world, it was the only note on the twenty-fifth.

A

They know we here Oh time zones right. Oh this.

C

Is it okay? There.

F

A

So I saw you using them. Do you use any sort of IDE? Do they exist for.

C

B

I was under using them for fur like in in the common line. Usually I use sublime text and it has a nice. It has some nice plugins for for Galang, I, think it in general during their problems with having cross references and go long and I personal I didn't find anything that was always working, so I kind of ignored the problem and just learn to live without it. But the basic plugin for sublime text worked pretty well for me, and it you know, has some some nice syntax highlight it.

B

Has it understand basic context and stuff like this, which works pretty well and, as someone suggested, the first step basic needs to be familiar with, go itself, but that's a pretty ambitious project as the first project in go. Bernie, I, I,.

A

Certainly agree, I would.

B

Suggest, starting with fixing some simple box just to learn the code base, I think this will be somewhere around cubelets, so learning and finding box marked as help wanted and and component cubelets. Probably is a good starting point. I'm not sure how many of them we have, but there are definitely some box without want it. South start with this and then once the code base is users developers more familiar with the code base and we've go it sells and how we submit PRS and how the review process goes. Then yeah, adding more more volume.

B

Plugins is I'm.

A

Going straight to I'd like to implement a custom volume, plugin is pretty big jump. Yes,.

B

Yes, yeah I, agree. I agree. Do.

A

You do you have any favorite go lighting references for somebody who wants to get started with go I know everybody would always told me Python or Ruby the hard way, what were fun references for just on warding on to Python and Ruby, because apparently they there's it's. You know their websites that walk you through introductions. Is there any kind of that kind of thing for go honestly.

B

I'm not sure I can just sharing how we learn this in words, because when we joined the project we didn't know you go either. So basically the first thing was there is a pretty next tutorial on go. Go lack page like the official page. We interesting example some nice corner cases. Then there is an interesting document written by someone from the team in month with you, I think. Was it dime sniff again? Maybe someone else I, don't remember. I could try to find it. He has on his github.

B

He has a simple page with interesting quirks of go and how you kept me misled. If you know other languages- and you don't know calling very well, there are some examples of what can go wrong and what can go very wrong. So those two combined, I think, is a very good starting point and basically then I would just suggest you know starting start coding. Fixing easiest box is wrong and.

A

You already plug the Help Wanted bugs those are great yeah.

B

Yes, yes, yes, I, think this is a good starting point to actually learn our code by the company's code base. Mm-Hmm.

A

Cool, let's see what else we have in our little Q and we may run out of questions, and we may call this an interesting start. I, don't know if you're interested in participating this this but I'm looking to try to start a basically a beginners to the Cooper Nettie's group beaters to keep Cooper nidhi's dead group so that we can have something like this. That happens more regularly than just once more sure.

B

Of course, I'm happy tub and.

A

You wouldn't have to just talk to me in this new picture.

C

That's also always great so I don't like.

A

Yeah, so here's one is couplets read-only port 10 255, enabled by default.

B

Okay, I would have to think a new point he somehow to the world. Where is this question sure.

A

It was during june twenty-fourth in my time zone at 11, 16 a.m. it's by any goodness.

B

I think it should be, but you can try. Yes,.

B

There's a fine variety: there used to be a file called you, or we can maybe try to find it by the port number. It would be maybe more interesting. Okay, that.

A

Wasn't too many.

B

Exactly surprisingly mm-hmm, so let's see what I get in this file.

F

A

To your point, also about help wanted to use is important. Yes,.

B

It's a very good point: cublas read-only port expose is basically running services from the queue blood, maybe of reasons necessary for hipster to collect. Mentoring starts from the kubaton keeps you can transition to using the ssl endpoint. Okay,.

D

B

See what this is used.

E

Okay, again, not that many culet options for me this is this is the one. So let's find this time and then only we'd only part and then we're to give you don't you further.

B

Italy program set to 0 to disable, so by default. This is enabled because by default will set, as seen here, we set this part to whatever we had in this other files that we just seen.

F

B

That means judging.

D

From the comment since this is.

B

D

It mean that this this product is enabled and it by default, gray, cool.

B

Another one uncertain there was nothing answered.

A

A

Let me see what I have here: we've got questions about the clas which I'm not gonna, although I did learn a bunch of that about that yesterday, not gonna dig into those well, actually, maybe we can dig into those. I know.

B

I will be able to answer, but yeah. That's right.

A

No, I think those are more specific than I'm able to help with either. Let's see what we have here,.

A

You were mentioning Daniel he's answering a bunch of a bunch of bees. Yes,.

B

He's pretty active facing oh.

A

And I've run out of Cooper Nettie's denim questions. Okay, so let's go over to Cooper Nidhi's users, because I bet the users have questions too sure.

D

B

There's something happening right now, right.

A

B

A

We want to invite them to join us sure.

C

A

Can they wanted fantastic.

A

E

I know what the right way to define an endpoint for services interested work, even when around them is this.

A

Let's see if they are want to come, join us okay,.

B

But I think that the user just realized is flat fletchling. There was some problem with co, Bernie's credential that were not sad but surprising. The cube CTL had a very stretch error to inform about it. Alright.

A

The syntax error, instead of actually giving an error it first it incorrectly. Instead of saying you didn't give us your credentials. Yes,.

B

Exactly exactly that's my understanding of what is happening here. Oh hey,.

A

Normally, it's time to get up mile arms going off.

B

That's still pretty early, I would say.

A

B

Like 530 yeah.

A

B

A morning person.

A

Let's see what else we have in here.

A

If the process of the pod crashes, it will be locked at the pod status me. This isn't a question. No, it's an answer.

A

In multiple PVCs use, one pv.

B

There is one about either a flag to cube for Eclipse it'll, get event to return. All events instead of the last x I keep running into this issue. Oh doctor just hang, oh.

A

B

A

B

A

B

Goodies thats wonder if I have any cluster running right now, so we can actually see.

A

Lots of flags yeah.

F

I can see that's interesting, but it looks like.

B

So this is not.

F

B

/, sir, but it's about get in general.

B

So one thing to know is that events are garbage collected at some point and so there's do not stay forever, hmm but I don't remember exactly like how long this day I think it's configurable, but I would expect that's enough to debug anything so, but unfortunately I don't have a running cluster right now, so that it would be hard to actually see something and take you too long to actually start running something.

E

B

hmm I would expect that this is actually listing all the events that we have, but I don't know, are.

A

You game to throw up a cluster and take a peek okay.

B

A

B

Stop sharing my window for a second, I will create a cluster and then be right back. Okay,.

A

That works and I will continue to look.

A

Or you can share, show shitshow sharing. You can share the creative cluster unless their secret information I'm.

B

Sure so there are some yeah just just a sake: no yeah, just a Jew key clutch because its fastest. But oh.

F

A

Whew I haven't question for us.

A

A

A

It's interesting that no one's taken us up on the office hours I. Suppose people will want to see what it looks like and try it and.

D

B

B

B

You, sir, you you.

B

Maybe in the meantime, we can jump to another question when the wild- yes get it I flagged.

A

Another one um I'm installed q brunette ease on top of docker on my OSX machine for development. Everything works great, but probably once a day, the API server no longer reachable on the blue. How can I diagnose what's wrong? I currently kill the API container, which is restarted and after that it works again, but that seems like eleni solution so.

B

First of all, I would try to read the logs and lots of the API server and see what is happening there. If anything.

D

B

Going on there, there should be some periodic log messages that will show that it up and running and if it is then I would expect. That may be something there is something between for the diagram networking, maybe on the darker side. That would be a gas, but I think that the first step is just to read the locks and see if this is still running and from that point we can, you know, understand more and see. What's going on it's hard to say something more concrete without the locks so yeah right.

A

Ya know babe sense, so the answer is check. The logs.

B

A

Few miles we have here.

A

A

I want to be notified when a process crashes and doesn't exit gracefully AKA downscaling organically. I love that phrase any chance to get any chance to get. If a pot is an error on the cluster, with some sort of metrics tool.

B

So, with each such event, queue blood should generate as I said event, then the user could watch for, and there should be information where it happened like on which machine what part? What was the target and what actually happens? I guess this is one of the ways that the user could be notified and.

A

This would be the same event that we're spinning up a cluster to take a look at and then see how many we can, how many historical events we can watch our extended different event.

B

And so I didn't get it so.

A

You're spinning up a cluster to take a look at how many events we can see to say condiments same same space. Yes,.

B

A

And they are garbage collected after some amount of time and we're trying to see if you see all of them before garbage collection or if you see only a subset.

E

The pleasures created it just needs.

B

E

Go make some events.

A

B

Just need to push the credentials yeah.

A

Okay and I will take a peek for a next question after we figure out that event, question.

A

The many faces of troubleshooting hmm I.

A

Have another question if you can think in tight at the same time, I have.

B

That actually I can start sharing. The screen is.

F

B

It mullis running so I created a cluster europe, west 1b and a grand new cluster I have the credentials something mm-hmm the one, and so when I get events. Indeed, there is shorter list of events. I can also at least event from all the namespaces.

D

B

This is much longer now less.

F

Actually do this.

B

Okay, so majority of events happen in cube system as expected, because there is nothing else. Basically, in this cluster, let's see we have something around Saylor's, just failed scheduling.

E

Created pulled and replica.

B

So, to my knowledge, this should list all of the events which are available.

B

But as I said, the event will be garbage collected at some point, so it it's not a never-ending storage.

A

B

Additional, like older events, will be starved by heaps, will be read by hipster and then they could be exported to some external database. I insult Iggy.

A

B

Logging or something like this so depending on the set of those older events, are also accessible in some way so again, yeah that that depends to see if we have actually an event where we can see. If something bad happened to container. Maybe we can try to kill manually one of the containers yeah and see what happens then. So, let's take let's.

E

See what now do we have? Maybe I can just a search to the gm's directly, we'll see.

E

To manic SSH, let's see.

A

B

Okay, we have something: has two ND, you probs yeah, but these are not regular. Okay, let me go back here. Let's run something.

A

Hello, there seems the hello world in Coober. Nettie seems to be hello. Engine X, yes,.

B

Exactly but then that's not a big surprise. No.

A

But the, but the part of me that works well. The part of me all of me that worked it into mix before coming to Google is always loves this and, like all the kitchen eggs.

F

E

You should have engines there, it is excellent.

A

E

We can stop.

E

In 23 seconds ago, so it seems it was.

B

Restarted yq black- I guess because the creation times much new work and the container ID is different right. Let's see what happened with event, this should be also in the description and.

F

B

E

Why don't you see this schedules created and started interesting?

E

B

Seems that there is no event about it,.

B

Which may be there should be, or I missed, something in this little exercise that we had here.

A

Yeah cuz there's like there's created with the original docker idea of for to be, and then there's created with the docker idea. D2D after you killed it.

B

Yes, that's a good point.

A

But it doesn't so it's scheduled created, started, hole.

A

And then create and start interesting.

E

B

A

Pole created started, it doesn't say hi.

B

Notice that there are multiple containers, this ID, then this ID right.

A

B

Is no information about we.

A

Found it the same that.

B

C

Cleanly killed it, you.

B

Would have to probably ask someone from cute led to action and what should I exactly do? Two for.

A

B

Would expect that there is an event saying that there was a problem of a complainer. The reason may be a feature request is appropriate, for this looks like a reasonable feature request. I would say: mm-hmm. Okay,.

F

A

You write, it might be that you said docker stop, so it didn't throw an error, something.

B

Like this that's possible exactly exactly, we can try again, and maybe you know, try do something like I'm gonna try to assess their again and maybe like kill, kill.

A

A

No, so this is a question for me, so the pod does the docker container doesn't die, but engine X inside it dies does Cooper Nettie's fix that if we just kill the engine, X processes.

B

Absolutely like container is basically a process which is properly configured basically the idea of having a container just having a process which is well better.

A

Eyes like isolated.

B

From the rest of the system of the rest of the machine, so I guess it should work. We.

D

Just need to know which process we want to kill and the.

A

Master will do it. It's.

D

D

Okay, let's go talk to say about it: okay,.

B

Okay- and he is restarted, we.

A

Started excellent so.

B

Let's see again now what happened with our God, because.

A

That was pretty unceremonious.

B

Failed saying error: sink parts keeping Feist to Senator, fail to start container mm-hmm Pakistan secondary starting failed container okay. So there is something about a failed container it's time. So this time we succeeded in breaking things. It's.

A

B

A

B

You break it and you get.

F

An error message.

A

F

B

Okay, so it looks like we have. We have an error, probably it's not too verbose in the sense that it's not obvious that there as well. There was actually a failure, but when you read it carefully, then it's kind of obvious there is like restarting phase container and jinx, both whatever and yes,.

A

D

A

We've got, we've got an answer and we built a cluster and I've got one more, which may be short or long, and we could decide if we want to answer it. Alright,.

F

A

F

A

Image pull secret was working previously double checked, it's still very valid credentials. Anyone have an idea about this error, so the error says: error sinking, pod skipping failed to start container for Clary it for Clary core for with an error image.

F

Pull find it: okay, fine I,.

A

Mean you mean not written my reading, it aloud isn't perfect. Oh.

C

I just need to find its woman.

B

There for me, if I see this.

A

Here I can send you, they have learned. Oh no I can't yes, I can. I can send it to you in slack. I can send you the URL and slack, I'm learning how this actually can work.

A

E

Where should you it.

A

Should have their am in the dumb lower from the direct message now further with little red tag is from me: oh yeah, there you go.

B

It works French.

A

Magic know: why is that malarkey, it's taking you back to mine, okay, let's find another way to give you this information I. Think.

B

Of Mexico knees, users and maybe well what was the other name. I was a search for these yeah.

A

Let me find it because now I clicked out of this, let me find my starred messages, because I think I started um so it's see our eyes ii ii, ra, IG, crees, craig.

D

B

So, to be honest, it looks like it's not a problem. We the secret itself, but to the fact that there is no talk. Yes, easiest thing to verify. This would be to log ssh in the machine and just run docker pool on whatever um I guess this image we. Actually we can honor. So it's a private one.

B

It is a secret, but anyway like to verify this directly with docker and then see if this works in darker, then it's definitely some problem with either configuration of this secret or just backing culet. But let's start with verifying the doctor is actually working like if you run directly this docker full and this image cool.

A

That's really helpful and this has been fun. Thank you great.

F

A

Time, hopefully we will, we will get live users wanting to comments, cus questions as well and we'll we'll see what we can do going forward. I'm.

B

Happy to help anytime, it's.

A

Been it's been great fun, we'll figure out more later.

C

I, don't want to wake. You have been fighting the money, no.

A

Five immortals: fine, fine, fine, fine, all right! Well, thanks to everybody who made it all the way through this recording and keep an eye out from when we do this all again,.

B