Red Hat OpenShift OKD Working Group, 13 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OKD Working Group 2021 04 13 Full Meeting Recording

Description

OKD Working Group 2021 04 13 Full Meeting Recording
more details here: https://okd.io
join the working group mailing list: https://groups.google.com/forum/#!forum/okd-wg

A

A

Hello folks in the chat is the attendee list link. So if you could sign into that, that would be great and um give everybody a few minutes to join.

A

And let me just see if I can pop into here, um we have a. um I thought you were you were in here as well. So that's great. um If people have items for the agenda um just pop them into the chat- and I will add them to this list, um but I thought we'd give everybody a couple more minutes to to sign to to log in and then ask vadim to give an update on the latest release and get any feedback.

A

People have and then maybe talk a little bit about what we're doing in the docs. If jamie's has joined us in.

A

By then, is there anything else that people have um for the agenda today.

A

Fcos incoming updates, I like that one thank you.

A

A

A

So I about one minute after the hour, I'm going to stop, sharing and um turn vadim. Are you there yeah all right, share your screen and talk about the latest release and then um we'll motor on from there.

B

Okay, um so last weekend we released another version of for the seven. It is marked as rejected, because our test suit has become much stricter and we didn't surely pick one of the machine config demon bugs, but you can still upgrade to it. It's perfectly fine.

B

We didn't risk and decided not to pull in latest changes in mcd. They have gone into the latest nightlys. That means the vsphere hardware version block should now be fixed and some more host name changes fixes have been merged in so that would need some more additional testing so that we could declare it stable and release it. Maybe this weekend or hopefully the um to begin following up uh if the test passed correctly um in other news, kubernetes, 121, rebases merged into 4.8, so and so did uh the fedora core s34 next branch.

B

So uh eventually in a couple of weeks, maybe we'll start cutting out the release candidates and uh get some more additional testing on those so that we could declare them stable.

B

On the problem front, the podman copy bug has been fixed, but we are still getting errors when we're using fedora cross the latest stable as a base image in installs. Because of the problems with extended attributes or something I will be chasing uh colin who's familiar with the situation to get some updates on this. um If I remember correctly, fedora 34 relays doesn't have this problem, so we'll get it for free. Once we get updates to f34 and some more good news. We have a new member in our team, helping us with okd.

B

Specifically, um please welcome mustafa, who has just joined redhead and helping with a lot of things related to community and stability. I think the official name is something like stability, which is exactly what we need to look at the front, um and I think that's pretty much. All I've got from my side.

A

Is that um that's mustafa um if you want to wave and turn your camera on just say, hi and um introduce yourself? That would be great.

C

Yeah sorry, I just uh find it more comfortable type while it's off yeah but hi, I'm mustafa, I'm. I visit berlin, I'm the first member in the new team, um I'm helping uh here and also in the storage team, so yeah uh I'm glad to be in nerdhat and hopefully I can help.

A

And, and did I um mishear vadim, I think I was on a call last week, an internal one where there's a number of folks coming on board, um similar to mustapha, to help um and they're going to be kind of interning with the okd working group, so you'll see um and we're getting some additional resources and one of the ways we're training them up is to bring them in via the okd stuff. So um that's actually really good news and and mustafa you're.

A

The first face to show up, but um I heard upwards of 20 new hires, um and um so this is uh going to be going to be interesting to incorporate them and have them understand our ci cd workflows and help us get some stability there and then they're going to be using that I think on the ocp itself, on the ocp nightlys as well. So there'll be some a lot of new new faces coming soon.

A

So that's um making me very happy and vadim even happier, I am sure, so you might get a weekend off yeah, maybe yeah. That would be. That would be wonderful.

B

That means no no kitty releases, so we'll see how it goes.

A

We'll get there we'll get there, but it is. It is a really good sign um um that we're getting some stuff resources out there and um yeah it'll be fun, so um welcome. Very much uh mustafa you're very welcome face so um yeah. So that's there, timothy, um I'm gonna, put you up next to um the update on. Unless people have feedback already on this latest release that are questions. um Maybe I should allow.

D

A

If somebody had a question, I heard that um otherwise we'll move on to uh update that's okay,.

A

All right, timothy take it away. I think you have to unmute yourself.

E

Yes, all right, uh okay, so a quick update on the progress in fedora qrs and what's coming next, um so something we've already mentioned. Is we are going to rebase the fedora 34? So, of course we'll wait until federal 34 is released for that to happen.

E

So the issue tracking that I'm just pasting that in the chat- um and yes essentially, I think this is going to happen well rather soonish after the after the release.

E

So the release, I think it's at the end of the month or something um and yeah third base- will happen a little bit after that, and the second item I have is that we are adding openshift machine config support to butane, so we've renamed the tool that was called fcct, the fedora config federacares config transpiler, we've renamed it to butane, and we've also added support directly into betaine, yes to generate machine configs.

E

So that's I'm posting the links right now, yeah. So the the idea behind the name is that uh it's uh it's it's fuel for ignition, because, essentially you can you convert your your butane configs uh to ignition configs. So that's the fuel you you put in your your engine to to ignite it uh yeah, so we're updating uh our docks already in federacares to mention began instead of scct.

E

That's going to happen soon, um so, what's important here beyond the name change is that we we had it in butane, a way to generate machine configs directly, so snippets of matching configs, which should help you.

E

If you want to do some very some complex uh root device or some complex, looks setup or things like that uh in okg kg, and that should be helpful. So this is, uh I think, it's done for for 4.8, so that probably won't work uh until photodate is released all right. So that's second item.

E

um Then we have two uh upcoming changes that are much further along the road much uh that will happen much later, but we want to raise attention right now about them to make sure that people can plan for it and do the necessary changes if needed so the first one uh which is planned uh for about two months uh approximately uh from now is the c group v2 change. So we decided to split that from the federal 34 rebase to avoid adding too many things and changing at once.

E

So in approximately two months, we'll switch federacares from c group cb1 to c groups by default, so existing node won't be updated. They will keep using c groups b1, but new provision nodes will start using c groups v2.

E

So here's the links here uh I.

F

See do we have to expect.

E

uh Problems with this uh switch that I don't exactly know uh it depends on. Oh no kitty. Oh oh, you managed that exactly. I think. Okay, we'll probably be um stick to see groups v1 until uh until testing has been done and that's probably up to volume to discuss that.

B

Yeah, I can comment on that we'll default to c1. However, the only switch is to remove a particular line in the generated machine config we tested this in ci and the only tests which failed were related to builds. I didn't look into those for quite some time, but since kubernetes 120, you should have the groups to be two supported in most of the pieces there. However, it's just not yet stable, so this is why we don't default to it. So this for the request. Change is important, but doesn't affect us right now.

B

E

Yeah, thank you. Essentially, everything uh in the stack is now supported. That's why we we made the switch so late, because we were waiting for all the pieces to be fully supported for for secrets. V2. um The the the unknown here is that I I'm not really familiar with regarding on how the hip date is going to happen, because, usually we well, I start tc groups v2 directly, so I don't know how to update from v1 to v2.

E

So that's maybe the tricky part for qd or to migrate a cluster for from v1 to v2, because maybe you will have to recreate all your containers, I'm not so sure about that. After that, you.

G

Definitely have to recreate the containers, because so someone someone ran into an issue not in fedor koros, but in open susa micro os, where I pushed a similar change to switch us to c groups v2, it opens using micro os and somebody had a persistent container instance that was just being invoked and on reboot.

G

It stopped working because the c group of stuff that it was pulling in for its settings were just no longer there and not available, and so, um if you were go from switching from c group v1 to secret v2, you need all instances that are saved on disk to be destroyed and recreated. Otherwise they will not work so.

H

That seems to be a hard requirement. She sounds like uh it breaks.

F

Things or isn't it.

G

Well, so normally a kubernetes system is supposed to handle this transition itself.

G

This is where, when you don't have an orchestrator doing that migration, for you like, if you're doing this by hand yeah, you can technically edit the existing running container instance and change all the settings, but, like that's such a pain in the butt that it's easier to tell people who are running you know podman or docker or whatever, on on a machine directly to just like just blow and re reinstantiate your containers, because that should be fine in a kubernetes context.

G

In an okd context, machine config operator should tell the rest of the operators managing the okd cluster, that instance that container instances pods need to be reconfigured, but because kubernetes instantiates them from pod definitions. Rather than leveraging the base container runtimes own ability to save container instance settings.

G

We may not be hit by this issue, but it is still something to be wary of if you're instantiating containers directly on the machine so from a general fcos perspective, don't expect your instances to survive the transition, but from an okd perspective I don't know of a reason why we would be in trouble with the transition um vadim. Do you have any reason to believe we would have a problem with that transition.

B

No, we shouldn't, because the change from c group c1 to v2 is effectively kernel, arguments change, meaning that requires a reboot and the old containers would just be destroyed, or rather the new cubelet would not be able to pick them up because they don't look as it expects them to be, so it would recreate them automatically.

B

But uh if you're running some quadman containers there yeah those would be dead.

F

But if you update to a version of fcos with this change, we have, and there was a lesson reboot.

F

G

Updating always requires a reboot anyway, yeah we've always been.

I

G

So my my guess here is that, because of the smartness or dumbness, depending on your point of view of how this actually works in kubernetes, uh it should actually be transparent to okay users.

E

Yeah thanks for the context, um so if I find my nose again yeah, so that's the group's v2, so that's coming in approximately two months so yeah after we rebased on fedora 34., um all right- and the last item here um that I have is uh fedora count. Fedora cares count me changes, so we're changing uh we're, hiding a way to to count the number of federacares nodes and how long they live. uh It's something that we're rolling out across all rpms3 based variants of federer. So it's not just federal course.

E

It's silver, bluetooth and uh iot, uh and it's um so it's not enabled yet in federal rest, but it will be available uh enabled in uh federal 34 for civil, blue and iot, so yeah we want to.

E

We wanted to give a longer heads up for pedro corres users, so this one is going to be available is going to be enabled at least uh not not before three months, so uh in instable. So you have three months to decide. You look at the change and decide if you want to uh to be counted or not uh so the issue- and the notes are here- uh essentially it's a very very uh it's highly privacy. Preserving it doesn't store anything, it doesn't send anything.

E

uh It just basically says to the federal servers: hey I'm a federal curse, node and I'm I've approximately lived this amount of time and it's very approximate you you basically cannot derive anything from that.

E

So yeah, it's it's a really really small, um counting method uh really really, and we we really care about privacy here so uh but yeah. If you feel uncomfortable about that, there's no issue at all and you can easily disable that it's just uh just uh a short, a short masking a simple unit and that's it and- and you can do that ahead of time too.

E

So if you do that right now in your notes, you know in machine config, you will never be counted so yeah, that's uh mostly it the all the details, the um how to disable that it's it's all written in links in in documentation that is linked into the issue. I've just linked and we'll make public posts on the chorus stats mailing list. We have a blog post on federal magazine coming up too. So this is going to be very public too, and all right- and that's mostly it for me- for incoming federer forest changes.

A

So, um just to be clear that the telemetry doesn't tell you that they're running the fedora core os and okd in any way, that's not going to be any visibility of that. What the source of the fedora coreos is.

E

Yeah, it's that there's nothing. That would tell us that you are running in okay. I don't think so. Is there yeah okay.

A

But yeah, sorry, it's it's just a account of how many not it doesn't even identify the source company. It's just very just account of how many fedora core os instances there are out there in the universe.

E

Yes, we we, we absolutely do not, then anything uh about the node. uh As part of the of the request is we. We essentially make only one http get request and uh what's it's it's. The only information that we're sending with this request is that we are in federal course, system uh using rpms3 and that we've lived approximately about one month, one week, uh four months or more than that and the federal version at a knee. So it's if it's better or 30, if we're better better. So it's really 34, but that's about it.

E

We don't send host names. uh Well, we don't send hyper address, but when you make an http request, the server has your ip address, but we don't send the the actual new ip address. We don't send any other information, yeah.

A

Looks good, that's it's great that'll be helpful, I'm definitely for for everybody who's in the in the community to have some visibility of that. That would be great any other questions for timothy um other than I think, maybe a blog post post. I think mike on the c group's v2 stuff, just so that people don't freak out when they hear it, because it doesn't sound like it's as big of a deal as it's.

A

It sounds scarier than it is, I think- and so maybe a short blog post about that, um because when people read that they may freak a little bit, but it was great to if we can test that early. That would be great and get some feedback on that. So then inform the ocp engineers as well. If something goes awry, that's a great use.

E

Yeah, the the federal correct, the sorry, the c group's v2 change is live right now in federal course, next, the next stream. So if you want to try that out in your notes, you can do that.

A

Okay, so uh so I'm gonna switch topics just to docs for for a little bit um and people don't have questions for timothy, just because I wanted to sync up with um bruce around his the taxonomy work that he was doing. That was just a gist file. The last time I looked um from the docs working group last week, um bruce developed a nice, a very nice taxonomy. I don't know if you want to share that again.

A

Did you get any further with that in terms of um because it blogging I I committed to creating a blog out of that um or are you in the throes of um exams.

I

uh Well, I am in the throes, but I did add a few more words at the top. You wanted about 200 words and I don't think I quite got there, but the that's okay.

A

You want to just share that with folks and maybe walk through walk through.

I

A

And then um I'll.

I

Explain what we're.

A

Going to try and do with this in terms of creating a a primer on for a checklist for people who are doing installs and deploys and.

I

Let me just find it.

I

I

A

Then I I did think I saw jamie on the call, so I'm going to put jamie on the spot next so get ready. Okay,.

I

So this should.

I

Be okay so there's the.

A

I

Should get you to it? Okay, do you.

A

Want me to share the screen and then um that make money sure, and then you can just explain what it is, that we're doing and I'll just share screen.

I

It's just a text document, so it's meant to be read with word wrap on. uh Otherwise, as you see the things go, sort of often to the sunset on the right hand, side there.

D

There we go thanks.

I

I

I don't know if you can do that turn on wardrobe from the raw yeah okay, so you should.

A

Be able to see the screen get.

I

Rid of mine now yeah so so, basically, the idea was just that uh a lot of people uh produce some excellent guides, I'm probably a little bit partial to craigs, because that was the one that I was first most successful in getting running. You know, but often you need to make modifications to them based on your local situation, and so I was, you know basically trying to pull out in a visible way all the choices that you need to think about.

I

If you were setting up a upi uh distribution because uh you know sometimes the choices are deeply hidden in the uh stuff, so, for instance, uh going with with charo's you pretty much have to get in and actually uh look at some of the scripts to figure out what he's doing and in many cases I think the choices that you make aren't critical to get the end result running, but you do have to know what you're doing you know once you get off script, and it is helpful to know that.

I

I think, because it takes a lot of work to put one of these, you know blogs or posts or whatever together and they get outdated. Almost immediately- and that's just you know the way technology goes, I don't think there's anything you do about that part, but but I think that if you're sort of aware of the underlying things that you have to do that that's helpful and it also might help you pick out the particular.

I

Guide that you're going to try and follow. So I was just laying out the choices and I I think it's like. I haven't really done anything in one way, because I think it's obvious to most people that have been through it. So I don't take much.

I

You know credit for just writing down the obvious and I I sort of first pulled out just to get it out of the way you know the the various uh you know cloud providers where you can uh use the uh infrastructure, provided you know and uh because in theory I guess that should be relatively automatic um and uh then went on to the uh the upi and uh the. So that's that's about it. The uh uh I think I've got the choices that are in anybody's blogs that I've been familiar with.

I

So if, if there's any missing, then just feel free to add them in the one thing that I didn't see. Much discussion on that I like to do personally is dealing with certificates, because the browsers are getting much more hostile these days to self-sign certificates, and uh since you can get a uh an actual domain for like 20 bucks for a dot ca. I think it's maybe 25 for com.

I

You know I mean it's fantastically cheap, then it's not that hard to get real certificates and because you have to have a wild card certificate, you know like start on apps dot, whatever then the let's encrypt and so on, is a bit more picky on who you are to give you that. So that's why that sort of interacts with uh having a real domain.

I

um But again, if you're going to have a uh a hobby set up, that's not going out on the internet, then you don't really care about that. So much uh until your browser doesn't let you connect to it at all which I can see coming so anyway. So that's that's about it in uh more words than I probably should have said so.

A

That's okay, so um what we were going to do is take that and turn it into sort of first, a blog post that other people can add on to so because the blog post, you can make a pull request against and merge in other suggestions and just using markup. So I was going to take that next step now, hopefully before next week and turn that into something, but if people had um other things um that they thought should be on there, just um you can once it's posted or be prior to it.

A

um Please feel free to layer it on, because what we're thinking that is this will end up not being a blog post but as we develop it and add, maybe links to little blurbs for each of those items and not full-on links to the total resources, but to the appropriate snippet that it will be one of the pages in the guidance that um jamie and mike mckeon have been working on so um and just another page in there.

A

But I thought we would develop, try and develop it out sort of in a collaborative way by getting people to give their tips on. You know where the best write-up was for each of these little things to get started, because I think the approach that he has here with this it's nice, because it's often you start down an install path and you don't think about everything else. That's coming!

A

You just hit the first install couple of steps, and you forget that you have like all these things in front of you, so this is kind of yeah. I like that that okd pilot's pre-flight check. So if you don't know what what all of these terms are, don't start um yeah, so maybe that's it, and uh so that that's that's, and so thank you for for doing that. We walked through it last week and I just thought we do that.

A

So if I turn that now into a blog post um and then post it to the working group and ask people to um to annotate it basically um with links that would that's kind of where we're going so cool and he are you a pilot bruce is that true.

D

I

But uh only uh private license.

A

I

A

Up to gibson well.

I

It's been a while, since I've I've flown to albuquerque over the rockies.

A

Nice, uh so um so that's where we're at with the checklist primer thing that um that bruce has so kindly created um uh jamie. You want to take over um and talk a little bit about where you're at and what the update is on the um the docs coming out of the deployment and testing workshop effort.

J

Sure so, the last little bit um I started on the weekend and probably will finish in the next day or two, which is basically taking all of the dependencies um that are necessary that are in charo's article and then putting them into the stub page. uh He's got quite a lengthy article, so it's taking me some time to get all the dependencies out uh once that's done. We are good to go all of the other.

J

Stub pages have been filled in uh and then we can just merge it in uh do a pull request or, however, we want to do it to get merged into the okd repo. Oh.

A

Awesome um you're making my day, so that's that's, really that'll be a really big push. I think it and a great outcome from that working group meeting and what we're doing um the other thing is. Next week we have a docs meeting scheduled. I am not available to run it um and I'm wondering if we want to take a week off um from the doc stuff um or if someone else could um host it and and that would have to be a red hatter. I think I think take a week off I'd like I.

A

I don't really actually get the time off, I'm in another session planning session. So um it's that's just that, but um if we want to have it I'm happy to host it next, okay mike, maybe maybe we will because then you guys can work through if there's anything with the migration that didn't work or whatever um we'll still have it, and I will give you the power to um create the blue jeans or I'll just turn the blue jeans on and hit the record button and go to the other meeting. um If that's that's good.

J

Yeah, you can either hit me up with the moderator code or something diane you can coordinate offline just wants to take the week off. Apparently I.

E

Can see joseph wants to take a week.

D

A

Yeah, well, I was just going to say joseph isn't getting the week off because um I'm going to thank him out loud in front of everybody for something he's about to do um and he is going to be our okd person on the openshift commons gathering at kubecon on may 4th he's going to be giving a talk that he is going to be recording um on monday this coming week and so we'll get that that up and there and he's going to be our voice and face there um for okd and also okd going to aro so azure.

A

So that's going to be yeah, we'll get. I got one little thing, so we're going to do that and if you haven't registered for the kubecon gathering, it's free so just join up there and the only other good news that I have is that they finally are printing some okd t-shirts for me and I I will have them available at kubecon itself.

A

So if you visit me in the kubecon okd booth, whatever that is that they've set up for me um yeah, then I will get you all links um to um get a t-shirt and self-order and pick the size and everything it took a bit of doing and unfortunately, joseph the um the version of the okd panda that you so beautifully recreated and flattened and made hipster cool, um didn't meet brand standards. So they made another panda, but it's still a panda. um And so that's the good news. um But I really like joseph's old panda.

F

I I expect to get a shirt from you again.

A

Someday someday before everything.

F

It's completely okay, it's a real life panda, but.

A

Yeah I uh I'm like yeah someday so um yeah, so that's that's the good news um and I'm I'm actually really looking up looking forward to kubecon um and joseph's talk and there's a whole bunch of other great talks too. That a lot of them is a. I just recorded an awesome one on podman and all the related tools um by some folks at ibm and work paid. There's just some really good content there, but kubecon itself is going to be very interesting.

A

So if you're there come visit me because I'll be the one person hanging out in in the okd booth for uh community central for a while, um I can, if anyone else wants to hang out there, that's a red hatter. I can um get you in there.

A

It might be bitter at times. It's not. We put a lot of sugar into the lemonade, so um there you go so cool um yeah. So that's what I thought I had for the agenda.

A

The other thing that we've been working on the docs or thinking about is really an overall docs strategy, and to that end I think amy is going to try and join and help us with that, because she's done a lot of amazing work with the openstack community, and so I'm going to try and use some of her expertise in maybe in you know, working with jamie and and bruce and other folks on um and mike with us on the on the every other week.

A

So amy he might have jumped off because she had another call at the half hour, but um she'll be there next week to to to tap.

A

So that leaves us with 30 minutes left or almost are there other things vadim or anyone else that you want to bring up or any issues that are burning from the kubernetes slack channel.

B

No, not really.

B

My only worry is that we would need a lot of testing on the vsphere bug and the fix we have in nightlys and since it's pretty elusive, it's rather hard to prove that it has gone away.

B

Other than that, I think we're uh in the clear. I also revived our relationship with operator framework folks who and next week will start working on the ocd specific catalog.

B

um I'm not yet aware of the level of my involvement there. Hopefully things will just start working, but that might take quite a lot of time, which means we would need some community um other than that. I think we're pretty good.

F

This uh bug you mean the uh hardware version thing.

B

F

B

Openshift is the unrelated stuff, because the fix is to effectively disable uh the offloading. I think um I'm not sure what would be the implication of that. So hopefully we won't have any significant regressions.

J

You have operator stuff, I just put a link to there. I don't know if that folks caught that, but there was a conversation um about uh convert and uh tecton pipelines operator that were added to the wish list. um A couple days ago and christian made an update. um So take a look at that.

F

And because of the respear thing, maybe we could invite the people that reported the bug. um If it uh works with the nightly, it would be great to get a positive result from them.

B

Actually this was the bug was very poorly managed and that's my fault. Honestly. It originates it from the gcp upi issue, which has been fixed by um opening ports. It's not reproduced with an ipi and then, since it had similar symptoms. Folks from vsphere uh have reported there as well. It took me quite a while to actually figure out that it's in fact a different issue.

B

A

So on the vsphere testing, is there an ask for I got to spell vsphere right? um Is there an ask for this? Is there a link to the bug and what to test test for.

F

Again, I I have a question mark because the f cos people released a new f course versions with hardware version 13. and before this bug was reported, they released, of course with version 15, which in fact is a problem, but currently, of course, is released with hardware version 13. It won't be a problem at all or shouldn't be a problem at all with new f um okd installations show. So we should.

F

I don't know how to reproduce it with new installations.

B

um Users can change the hardware version on their own and unfortunately they are very inclined to do so, because this fierce csi requires something like 15 or maybe even 16., that leads to people installing things successfully, then changing version trying to apply csi drivers and network suddenly goes down which is very complicated.

B

It complicates the debugging a lot.

B

But apparently it's very easy to reproduce by just changing the version and installing things with openshift, I'm gonna write up a little.

J

Vsphere testing protocol for this real quick, I mean it's basically just a paragraph and some links. Yes,.

K

Unfortunately, it's not one that I can test, because I'm stuck at hardware level 13 until we actually upgrade our physical hardware. So this isn't something I can even test.

A

And you were the one I was going to just ask. If you could, I would.

K

F

A

Person for that.

F

Very complicated bug, but does it uh if, if this fix, does solve things for vsphere, does that mean that also uh fcos people will switch to uh how the version 15 again is? Is it related somehow, through this bug, no.

B

Okay, I think it's fair for initial fedora coral scissors to stick to the lowest possible um version so that folks on 6.5 could use it um and users can upgrade the version on their own if they have to, but it's definitely not related to the current issue. It's actually just a great coincidence that we're starting with the lowest uh version.

A

Okay, so um so is there anyone on this call that can test this.

F

I have never seen the problem. um I we are just stuck on uh oving kubernetes and we had. We were not brave enough to change the network plug-in now: production cluster to openshift sdn in okd 4.6.

F

But if, if there is a version 4.7 with openshift sdn working on vsphere and we will try to migrate, a cluster have followed 5 cluster 2 4.6 with openshift sdn, and this fix. If it's available.

A

Is is this something that um that we could maybe get you to put a short email out to the group mailing list just to get people awareness of this to the other vsphere's fuss things? And just if you really want feedback on this book, I'm just trying to think of a little out of the box here rather than just asking in the working group, because there are about 300 people lurking on that okd working group um mailing list.

B

Right, that's that's one of the options, I'm just thinking that it won't be very easy to gather feedback on that. I think I'll start with poking people at openshift devs like channel, because the feedback loop would be shorter and if we don't get anything we'll start with the mailing list.

F

I at least you could um maybe you could post a link to the issue and ask people to um to comment it. I think it's worse. Try.

B

Yeah good idea.

A

Yeah and that and then just ask them for feedback on openshift dev. If that's where you want it um just come over there, um so yeah! So bruce you sort of sound like you have two, maybe volunteers, bruce and um and joseph, but it probably takes more than just the two of us two folks to do some real actual testing. So not that you're not worthy.

A

Is there anything else that we should be um raising up here, anything else, anyone's working on that's exciting.

K

I have a question: um I raced it up on the dev and I didn't get a response, but maybe nobody does that um so I'm playing with istio 1.9.2 and one of the things that I seem to have found is that it doesn't like um config, uh I'm sorry deployment, config stcs. um Has anybody ran into that? I had to convert a bunch of stuff to just deployments, um just wondering if anybody's seen anything like that with istio, in particular,.

F

I I can tell you, um john, that we were switching from deployment conflicts to deployments and with our customers just to be more to kubernetes and- and I I didn't understand- if you want to have an experience report of that, but it's a matter of a few seconds to change the determined convict to deployment and the difference is that you have to restart on every on every push of your images. You have to restart your your um containers, but you can mitigate that with.

F

If you use um git ops tools like auger cd, we use aggro, cd and augo cd can help you a lot in restarting deployments. If your.

K

F

K

F

To know jenkins.

K

For a lot of our deployment stuff but yeah.

F

K

Odd, because I I I probably have about 150 deployment configs and going through and uh changing those is just it's going to be a pain but.

F

I was just wondering.

K

If somebody had any experience with it,.

F

K

F

You need something that restarts your containers and, if you use, I can heavily.

F

Propose to use argo, cd or github's tool like that. That changes your images, image tag and, obviously you will roll out your new deployment and restart your containers with that.

K

All righty, like I said I figured give it a whirl, because the istio group didn't have anything to say about it either. So I don't know if deployment configs are just not supported in istio, which I would find odd, um but maybe not since it's not kubernetes specific.

K

So anyway, that's all. I got.

F

I think a default thinset view openshift or okd versions is to use deployments, not deployment configs. I read something about in that on that in changelogs that the new default is uh in openshift our deployments and not deployment conflicts.

K

Yeah, we use a bunch of templates so but uh yeah it's just one of those things about you know moving to a new system and trying to use new technology. You have to re, reduce some of our work, but you know that's probably a good thing in general.

K

Thank you, man.

A

All right cool anybody else's problems we can solve in the next 10 minutes. um Otherwise I might give you all 10 minutes back and um we'll we'll get um mike. You set up to run next one's next meeting.

F

I would propose to to block all um 1.7 releases until the next release if it works and um has solved all problems um that we were suffering uh since a few releases like this hotter version thing, the the um ip tables problem with the routers and so on. I would really like to re, propose to block all intermediary releases if that is possible, so users that want to upgrade from 4.6 get a fine 4.7 release that works out of the box without every any surprises.

I

Yeah well, actually, if we've got a few more minutes, I could comment on that as well, uh because the uh uh like I just saw this morning, uh somebody was looking for uh somebody who's asking for a long-term support version uh which christian properly shot down, because this is self-supported anyway, but uh it could be what the person was actually trying to get at was that a stable stream is really stable, as opposed to a testing stream or next stream, or something like that uh and from vadim's comment.

I

I could understand that it would be complicated to set up these other streams, but I could interpret what joseph was just asking for as a stable stream uh uh and uh so any anybody. That's wanting to actually use this for real, as opposed to just a testing lab, uh doesn't want things to have the potential of breaking whenever they're doing a new version, because you know you can lose like two weeks and reinstall and redeploying everything from source, which is a bit inconvenient.

F

From my perspective, what we have currently is is a fast channel and not a stable channel. We we always in every version of okd. We have a point that we have a stable version, but some since 4.6 we have lots of not so stable versions. I would expect more in a fast channel. I think a stable channel was really community approved. Stable tag would be a great idea.

B

What does when you say stable? Do you mean zero bugs? Do you mean it's installable and upgradeable or not.

F

Not zero bucks, but something like the hardest problems that block you completely are solved like we.

B

F

B

We have tests to ensure that we know that vsphere and aws are installable and upgradable. It doesn't mean your particular setup will install, but there is no viable way how we can do that. I mean.

E

I can understand.

B

A custom version plus open shift sdn, but yeah that would explode our metrics by 5 million ocp has nightlys and we only block on two jobs. The others are informing, mostly because we won't be able to keep up with the level of testing. We need to run.

F

I may be an idea: it's an idea that if you get the feedback from enough people, that this version is absolutely fine, that we mark it as stable and and.

B

Means that means we won't have the stable version in months in in maybe years mostly.

C

B

Maybe that's great, I mean I'm fine with that. The problem is we have to support the existing stable version which we have declared previously stable. That means.

E

Somebody else just believes.

B

His work- I am not setting up this ever.

F

But in behalf of state we have a rolling distros, that's absolutely okay, but some maybe it makes sense. If enough people say in this rolling district, we have one channel yeah. We mark some versions as more stable than the others and they go over to a different channel which gets not updated.

G

Every two weeks we're getting to the point of talking about now: we're going to have checkpoints and stable release tags and like yeah. No, if we do that, like the first of all, fedorkorios people will be very angry with us.

G

If we start doing that because they don't, they are trying very hard to kill that idea flat on the ground and the ocp people will be angry at us, because now we are we're going to create a problem in which we are either holding back things, but they are themselves releasing the production which at creates all kinds of weird questions about what is upstream side stream downstream.

G

What's the release cadence what's going on here like it, is not practical for us to move like that, and it is also not practical for us to suggest that we can move slower than ocp itself, because at the end of the day, they're the folks making, like the majority of the code, that that makes the okd thing.

G

um If, if anything, if there is a problem with your use case or whatever, and you don't have a way to make sure it stays fixed, then that is a defect in the testing apparatus and not a defect in the code base because fixing it once and then having a test to make sure it doesn't break again.

G

Is the whole point.

F

I I don't know if it's, if my, if my wish is something visible, but maybe all people that use okd in production. Maybe you understand me that it would be great to have some kind of tag that assures you that yeah um lots of people say this version is absolutely usable, because I have a few situations with my team that we were.

F

Maybe you can follow my chat conversations um that we had situations that were worked great with okd, but also we had uh upgrades that were absolutely mess.

B

A few runs, I can perfectly understand the goal to have the stable release and we have the stable release. The question is: how do we achieve this in a feasible way so that we wouldn't have to wait for somebody on ptos to to get us the final approval to actually release it?

B

I

That would mitigate it. That wouldn't be that terrible, um like I know that, uh like some, sometimes something that is totally silly can take your entire cluster down in a non-recoverable way. uh So, like uh the like, I lost my test cluster because of the uh uh non base64 encoded, uh uh password, uh authentication part which had worked up until then, and so certainly it would have been really because it had worked through many upgrades. It would have been nice if the release notes had said.

I

Oh hey by the way this is going to kill your system. If you don't fix this first, because uh you know once system was going, the mco would not pick up the new correct, full secret. Okay, it just kept persevering, and uh you know they uh like on slack. I, when I put that up, you know vadim said: try this okay, I tried that didn't work. uh John said: oh yeah! No, I had to rebuild my system and okay, so fine uh yeah.

B

That's the squeaky part.

I

World of hurt, okay, this.

B

Is this is a tricky part and we effectively cannot automatically test it, because all our ci systems have to report back so that we could collect the meaningful stats. This.

I

Is a very way of of going back? Okay, like uh you know, I haven't, found any approved way of regressing and upgrade the previous version so that you can try again.

B

uh Restart you can restore from the backup, so it's the only way to roll back the upgrade.

I

F

If any host os upgrades are a problem.

B

The host apps os is uh reverted by our pmos 3 rollback command, and you find this too correctly when, if your system has been working perfectly, when you do a net city, backup mcd would notice that you actually should have put it in the previous deployment to do that for you. But that assumes that mcd can actually do deployments. But um at some point you probably might have to help it and do things manually.

F

B

J

I

J

I

B

Sorry, you were, you were.

I

B

I

The backups, the backups, are manual right.

B

I

B

uh That's a feature which you.

I

Would fix right many other systems do an auto backup before they upgrade.

B

That's the tricky part. We only have a cluster, so when you make we can make a snapshot, but there is no place to store it, because all you have is a cluster. It has to be stored somewhere else.

B

You can save it on the disk space and, if you're and if your node goes down- and I don't know we corrupt the the file system, then this backup is unrecoverable. The backup.

I

Has to be stored outside.

B

Of the cluster.

I

If your cluster burns down because of a fire, I mean at some point, you're not going to be able to recover, but uh in the less catastrophic thing, yeah.

J

I

Catastrophic thing, then, that would help.

B

It just gives a false sense of security that yay we made the backup. Well, don't worry, but actually it's stored down there. So it's it has to be scripted. Probably we should stress that more in the dogs that make backups it's a critical part, if you want to roll back.

G

We keep going down this rabbit hole. I just have to ask something really quick. Do we actually have a way to like show people like how many people are on a particular version of okd or something like I? I vaguely recall that we have telemetry. Do we have like something.

I

G

Graph or something that we could put on a web page somewhere.

B

A

Yeah, it's all private background stuff.

B

Also we can. The problem is that only folks who are using red, hat's, pull secret are signed in. So this number, which is around 500, now, doesn't tell us anything. It doesn't tell us how many are outside, so the count mean from fedora might give us some ballpark number, but again it very hard to say.

G

Yeah so then we should, should we ask to figure out a way to leverage count me to say this is an okd f cause node and count those and whatnot like? I don't.

B

G

Something if someone, if the actual ask, is what it sounds like is that somebody wants to know how many people are on a particular snapshot version whatever on a different type of infrastructure. That should be a question we can answer.

B

We can answer.

G

That using telemetry.

A

But not with campaign it's time to break out time out, so I would like to move this into either the the kubernetes chat channel um or onto the mailing list. I think it's good content, um but I do have to kill you all and stop this um this meeting now. So I can jump into another one because I can see the next ground of folks are joining, um but you know it's definitely. You know it the more testing we can do the better. I'm just gonna the.

F

Mess the messages.

A

F

Earning was this: we are earning money with okd currently and we are switching to openshift because we have lots of business critical applications running on okd. That's perfectly fine for us, but it's absolutely great for getting your feed on kubernetes cloud technologies and that's why it is worse to get yeah. Okay in your in your company.

A

Yes, we all want you all to make money any way you can, and we all want you to test. So, thank you all we'll do this uh we'll have something on the 20th on the docks joseph I'll talk to you again. We can talk.

I

A

This in your talk on monday that you're recording so we'll get we'll get there. Thank you all for coming today and um always a pleasure.

F

Thank you, bye, bye,.

F

F