Cloud Foundry Community Advisory Board, 16 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Foundry CAB call recording 16th Dec 2021

Description

Recording from the Cloud Foundry Community Advisory Board call held on 16th December 2021. Discussions centered around #Log4Shell mitigation and general community updates.

A

Hello, everyone welcome to the december edition of the cab call um before I even realized that the year has started. We seem to have come to the um end of it, so I hope everyone had a wonderful 2021, definitely better than 2020. I guess but yeah here we are and as everybody knows, this is the community advisory board.

A

The aim of this call is to basically get folks to interact with each other about you know, general community matters pertaining to cloud foundry and as the community stewards at the cloud, foundry foundation, um representatives against me and chris, maybe we will share some details from the community, um or rather the governance perspective of what's been happening um with the cloud foundry. So, generally speaking, we just concluded a board meeting yesterday, and things are looking very optimistic for cloud foundry in general and specifically the cloud foundry community.

A

Our wonderful technical oversight committee has been hard at work, creating a lot of new and interesting ways in which people can contribute as a community and come together and push. You know the limits of cloud foundry forward in general.

A

There's uh there's been some interesting developments um in that regard. So all our um working group um maintainers have been marking a lot of issues as good first issues. So if there are folks out there who are listening, who want to get started on contributing to cloud foundry, I think it's a wonderful place to um get started and start making use of these issues marked as good first issue in order to you know, pick things up and work with them.

A

So another important change that I thought I should talk to folks about is what was formerly known as the community management working group is now the service management working group and I'm dropping a link in chat for those folks who are interested in learning a little bit more about the service management.

A

Team, so this has been an update in what they are responsible for and how they can work with that and it's, I guess, it's a bit of a fork from what was discussed originally, but it makes a lot of sense to keep this going and work with that in general.

A

A

There's also a big push for all the working groups to adopt an open road map so going forward. Everybody can.

A

Start both consuming and contributing to the roadmap of each of the different working groups and have more of a say in steering um the way working groups are putting out solutions in general, so yeah. Those are really um the the big um sort of updates from the community.

A

So I guess we can move on to the big blog for j topic unless anyone has any questions or want to add anything.

A

A

Much like the rest of the world, I guess the cloud foundry community has also had a busy weekend. um Unfortunately, it had to happen uh before the holidays. When you.

B

A

Everybody is in like a celebratory mode, but at least it happened when everybody was in office, I guess and um not exactly celebrating so um for those who need a little more context.

A

There is a remote code, execution vulnerability that could be exploited on the log4j tool, so anything that was any version of log for j before 2.15, I guess was identified as being vulnerable and it was a very critical severity and so.

A

The cloud foundry code base is not an exception to to, I guess, a lot of other software. We are also we've also been using log4j in a few components, and we specifically identified like five areas where we needed some urgent um efforts to go in and and do the fix so mitigating this is, you know, fairly straightforward.

A

You had to update the version of log4j that you basically used, and after we had heard about this- which I guess was on the 10th the friday, the security, vulnerability management working group so to say, that's a mouthful, but basically there is a working group that is responsible for voluntarily vulnerability management, and so those folks, you know, got on board quickly. They ordered some scans to be done, so there is a tool called app check and basically, the results of the scan pointed out.

A

Five areas, like I mentioned so uaa, which is the authentication tool great hub, which we use for credential management and rotation c of deployment itself made use of log for j, and then the newer cf4 gates was also making use of log4j in some cases, because there was significant overlap with, I guess, uaa and gridhub, but there's also a couple of build backs. uh Specifically, obviously the java buildback, I guess, but also the php buildback, um were- were declared as affected.

A

So after you know, we got this list and made sure that appscan completed its entire protein, which is basically coughing up uaa like multiple times and stuff um by late evening on december 10th, we had this set of um areas where we definitely needed to go in and cut a few new releases and things.

A

So there was some information that we got again later on um friday about this being slightly more multi-dimensional than um than what it seemed. It was not like as simple as just you know, go and change versions, and things like that, so um there so a couple of different ways in which we could fix. This was what was discussed and uh recommended, and later on, saturday uh was when we started to think about cutting our first releases and updates, so the uaa team actually released the first fix.

A

This was early morning on saturday, so if you're using uaa, I guess the simplest way to get over, it is to upgrade to version 75.11 or higher and then.

A

On monday, early, like rather late on sunday morning, but for practical purposes monday, I guess cred hub released the 2.10 version, which mitigates uh which mitigated the the problem for grid hub. This was followed by um caa sorry cfr, kids having a simpler path, because uaa and credit have both had new releases. Then cfor kids was just bumped up to use these two new versions on monday, and we had like um the problem solved in like three four areas uh in terms of communicating this.

A

uh The security team also put up like some advisory on the cloud, foundry blog and we were able to push this news through twitter as well. Now.

A

On wednesday, I guess yesterday um somebody had reported that it they were hearing news about 2.15.0, also being vulnerable, um so it was recommended to bump up log4j to 2.16 and then all of the different components um are now updated to 2.16.

A

So everything is through the pipeline, um great hub, uaa cfr gates and some of the build pack stuff. So I guess we're yet to hear about um the java buildback, but otherwise all other components um seem to be updated and the mitigation is is declared complete.

A

As of now so that's been the story from the cloud foundry community site about what we've been doing so far to fix this. If folks have any other experiences or stories or anything they want to share, um you know please come forward and make use of the time it's as much yours as it is us.

B

So um I can share a little bit about the cloud.gov experience. If uh folks would find that interesting, um I'm not sure how other people did, but um I first heard about this friday morning, u.s east coast time, and even though we had no evidence that we were being attacked or were vulnerable to. It immediately declared a potential incident.

B

Since we knew that uaa and um kredhub and um some other components we run might be vulnerable and it was. um uh It was gratifying that we have tools um within our deployment process to use operators to um set the initial property value that you could pass to, jvms, that the dash g switch for log 4j dash no format message lookups or whatever.

B

So we were able to get ourselves into a um a pretty good position um before the end of the day on on friday, as well as having having wife pools and then I've just been very um pleased with the speed at which the um the patches have come come out of the community and and the attention that you all have paid to it. So yeah thanks.

A

A

Anyone else have any thoughts stories, comments, feedback.

A

Okay, well thanks everybody for joining uh the december edition.

C

Sorry, sorry, sorry I was, I was scrambling for the mute button.

C

Sorry uh so I'll say uh I used to be on the cloud because team, but now I'm a cloud like a user for another government team. um It was gratifying for me to be able to you know, knowing that we were on a standardized platform that we could look at ourselves in the community and see what was going on, and I was able to bring that the attention to the cloud.gov team about that operator. Environment variable group setting.

C

um So I could, you know, help other people, so that was great and benefited platform. um One thing I would have spent more effort on the cf side was the the php build pack, um the php pack, uh you know issued a a new version because of app dynamics. um App dynamics is a service. That's only available, I think on mostly on pivotal deployments or sorry vmware deployments, um it's generally not applicable to the to anybody running the open source version.

C

I think most of them are not don't have an app dynamics service on their platform and it was the driver for that service that was susceptible, and so I think one thing that could help is when, when they're, when releasing updates for things like that, be explicit about what might have put you in a category to um be susceptible to the vulnerability, because when I saw it you know, I do have php applications that were not on my radar at all as being affected, and when I saw there was a php build back affected.

C

I you know I went into high alert, saying oh my gosh. I thought I wasn't affected. Maybe I am um so I think you know the release notes for remediations, saying not only we've remediated the vulnerability, but also this is how you can tell if you were um if this might have affected your application, because in this case it was an optional piece of code that was not invoked unless you have an app dynamic service available and deployed and bound on the platform.

C

So you know, if you, if you just say, hey, there's a new php buildback to deal with log4j people in incident response positions are now scrambling to say: oh crap, what happened? Whereas if you instead said there's a new php build pack uh to deal for a log for it to deal with log for j.

C

But uh if you weren't using app dynamics, then it doesn't matter to you that would have helped a tremendous amount. um In my case I was I was curious enough to to go. Look at the php bill pack source and say what did they fix and I saw that it was academics, and I could look at the code and say: oh it only it's only invoked if there's a service bound.

C

So um chris chris is saying. Is that the case?

C

And the answer is yes, as far as I can tell, and so that that's important, because um you know the the nature of which component or build pack is updated, will determine a whole swath of uh of incident response process that cascades down from that in the organizations that are that are using those things and a lot of people won't be as curious as I am, or have the time that I that I did to go dig in and figure out well like was I actually affected, based on the change they made as opposed to well, I used the php build pack and there was an update, so I must have been affected um so that can that can trigger a lot of scrambling and running around, and so thinking about you know whenever you, whenever you give a notice to somebody saying hey, there was a problem.

C

You also want to give them information, hey, there's a problem, we fixed it. You also want to give the information to say and here's how you can tell if you were affected, um so I would suggest that, as just a general security practice for any kind of security update uh notification.

A

Yep- and I really apologize for not having considered that as part of any of the updates that I personally put out um in in any of the communications that I wrote on behalf of the community. So.

C

No just be clear: no apologies necessary. It's fine.

A

C

Just you know for future uh something to consider as sort of a standard process when you're, when you're making these announcements.

A

Absolutely, and and even in specifically in this case, I think that's a great point that you made and uh yeah totally agree with you in terms of also communicating the scope of what gets affected. But it's not so. Thank you for bringing that up.

B

uh So I'm just going to jump in with one other thing that um uh brett very helpfully over the weekend uh brought to attention to our attention that tweet that I just uh put in the chat there that there is the ability to set the.

B

Set run environment group variable so that gave us the ability to um across. That would give an operator. I should say exactly what what we did or didn't do, because I haven't been cleared to speak on this, but it would give the opportunity for a an operator of a foundry to set a variable that for log for j versions, between 2.10 and 2.14 to not um by default parse, um the jnda lookups, and so one could um set that across a foundation and then in the course of a deployment when cells get rolled over.

B

All of the tenants will have the applications restarted and should pick up that environment variable um particular developers are: do have the opportunity to override that variable if, if they need to by setting their own environment variable and but it was just a a nice tool that operators have in their pocket for making changes across the platform in a minimally invasive way to um secure their tenant applications.

B

So that was cool.

B

It doesn't solve everything because you know you still need people to uh restage their applications and check their dependencies to make sure that they're not holding a log4j someplace else. But um you know every mitigation that you can get out quickly sort of narrows the the opportunities for uh bad actors to to get in yeah. I I.

C

Think um uh on the on the environment, verbal thing, I think that the tweet I was actually quoting or passed over had said something about like you know, you have to set this as an operator and then ask everybody to roll their apps. It's like! No, you don't really have to ask everybody to roll their apps, because the platform contract basically says we might evacuate you from a cell and therefore your app is going to get restarted and we'll pick up environment variables.

C

The only reason they would need to be restaged in this case um since the since the variable affects runtime behavior, they would only need to be restaged if the environment variable you set affects staging behavior and actually the platform has been a little overly cautious on that in a lot of documentation. It says: hey if you said environment variable, you have to restage your app and that's not true. You only have to restage your app if the environment variable you set is used during the staging process.

C

So, yes, that's a blanket statement you can make. That will make sure it's always right, but the reality is, if you said, environment variable and it's not used in the staging process. All you have to do is restart your app to pick up that environment variable, and so for that reason you know you can set it as a as a as an operator and no restaging is an necessary. We know this variable is only used uh during the running period. It's not used during the staging period, so they don't have to restage.

C

You can just you know, roll roll, all the cells and know that your customers apps, have all been restarted and they're all picking it up.

C

So I really appreciate the fact that the platform allows you to minimize the impact on customers that way, and it's it's a huge it's a huge deal and I think um we should all be trumpeting it from the clouds uh in blog posts everywhere to say this is why you should use a pass, and this is why cloud foundry is good.

C

So I'd love to see a blog post following up on that and saying you know how being how using a path helps deal with systemic or widespread problems like this and here's an example of how you know. Mitigations roll out.

B

Yeah, particularly because um web application firewalls are so limited in their um utility here, because there are so many different patterns um that it you know, no regular expression was going to catch them all and um any particular I mean it would certainly catch. You know protect you from the spray and pray people out there that are just saying: hey.

C

B

What's uh um uh targetable out there, but you know any um one who was particularly wanting to target your organization would easily find their way past a uh web application. Firewall is my understanding, I'm afraid I'd have to leave in a minute, because at another meeting I couldn't reschedule but um I'd be curious of other people had experiences that they wanted to share on uh responding to this before I drop off.

B

Oh, I never introduced myself, I'm peter burkhold, I'm the security and compliance lead for cloud.gov over in gsa.

A

Thanks for joining us, peter and and sharing all that information, um the folks have anything else to share anybody's mics on mute by mistake.

A

No, I guess not again. Thank you so much for joining us in this edition's cab call, let's all meet the following year and thank you ashley for patiently recording and helping upload. All of these calls and keeping the tradition of cab calls alive throughout the year. um See see you all next year, then bye-bye.

D

Hi everybody thanks for joining happy holidays and happy new year.

C

Thank you youtube bye. Thank.

D

C

Thanks ashley thanks.