Cloud Custodian Community Meetings, 13 Jun 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Custodian Community Meeting 2023-06-13

Description

A few topics from this meeting:

- Review initial work on the Oracle Cloud provider
- Docker build failures
- Our reliance on the Azure CLI

You can find the notes for this meeting on our github repo: https://github.com/orgs/cloud-custodian/discussions or at HackMD:

- https://github.com/orgs/cloud-custodian/discussions/8659
- https://hackmd.io/@c7n/ryyiU-Lv2

Check out our Slack for more info! http://slack.cloudcustodian.io

A

All right and we're recording now welcome everyone to the June 13th Cloud custodian community meeting I'm going to drop the meeting notes the agenda in the meeting chat here.

A

Everyone has it and then I'll share and we'll talk through it a bit. uh The first section is, as always, is the intros. So if anyone New or Old to the community meeting wants to jump in with some introductions, uh Now's the Time.

B

Yeah maybe I'll go. um My name is Robert Campbell I work with Oracle I'm based in California, the Bay Area, um so we've recently put together a provider for oci cloud custodian provider, and so we're just going through the uh the process for getting that integrated into the builds and aqua sodium builds.

A

That's good if anyone's interested in the work that goes into adding a provider that pull request and I should actually share that one that there's just there's so much good work there and uh and documentation there's a lot of really cool stuff in there. Let's pull it up.

A

B

C

The only thing it was it was failing on CI because it could look like I had the oci provider SDK installed.

A

Oh okay, I didn't see you come in there, hello, I didn't know. If, uh if you were in the midst of a reinforced stuff because I know, that's that's online.

C

uh Pull out and come to this nice cool.

A

uh So Robert was just doing a little intro uh welcome glad to have here uh anybody else, introductions.

C

Ryan yeah there's.

A

D

Sure um yeah I'm Ryan um I work at Capital One. um This is my second meeting um and yeah I'm working on some SES stuff to support one of the services we're launching.

C

I think that just got merged FYI. Oh.

A

Did it? Okay? Okay, oh nice, cool.

A

uh Cool welcome again Ryan glad to have you.

B

A

Intros again, there's no no pressure for the intro or for the camera. If you do intro.

E

Hi I'm mandine: this is my first meeting I. Have you Cloud custodian at my um last year, I implemented a bunch of AWS policies. This year, I'm working on Azure and I've had a lot of fun working with um this code base. So I'm happy to be here, cool welcome.

A

Happy to have you here and happy to have more people using the attributes, I think I think those get exercise less often than the AWS stuff. So it's good to have used questions.

B

A

uh So cool thanks for joining.

A

A

All right come back to intros later if we have any more okay, so some of the agenda items for uh we've got some fixed, uh some things that are still in here from from previous meetings, uh still a note about moving on from python37 and about the the tips and tricks section in discussions.

A

um I think there was actually there was some slack discussion recently that I think might make its way into discussions around um and Mandy I wonder if this might interest you or, if you're in Azure, if you've been using custodian for Azure, do you also use teams.

E

A

Use teams: okay, and have you been using custodian to send like the web hook action to send teams messages, not.

E

A

Yet okay I find it that that's there's a couple examples in the docs around using uh sending out to teams, um but it's not uh it's not as fleshed out as some of the uh some of like the slack integration. I think the tips and tricks might be a good way. We've had some slack discussion there, uh some ways to make that stuff a little clearer.

A

uh But if you come across anything that seems under documented or inaccurately documented on the Azure side uh would be cool, even if it's just an issue to let us know we're jumping into slack. That would be good, so we can help that okay.

A

Okay and then uh policy packs, so this was Darren's point I know Darren's not here today, uh but in the last meeting he was talking about just having to be a repository of examples, I think uh Kapil. You may have mentioned creating them per provider, but under the custodian org.

C

I I think the follow-up was for Darren to create a GitHub issue as a proposal which I don't think he ever did so I think I paid him about it, but um Colton way on that, but and.

D

A

D

C

Have that discussion on issue.

A

Sounds good? The only other agenda thing I know that we've got uh there are Docker. Docker builds, are failing right now and.

C

I also wanted to add in there was a question about the Azure CLI, like the the fact that we depend on the Azure CLI for the for getting information. It wasn't clear because, uh on that one I wanted to just walk through. Why? Why we couldn't just read the follow ourselves and said, depending on the you'll, see a lot.

A

Yeah and actually I wonder if um I not that I want to we'll command you to the meeting and ask her a bunch of questions about editor, but um there was it looked like we were. Let me let me pull up the one. uh The pull request with this.

A

Okay, so that's 86.32.

A

And this one, what happened there was a. There was a change to the Azure SDK, where it added it. Had this uh internal method, a run command method and they added a required timeout parameter, and we weren't using that. So so some calls started breaking and it looked like where we were using that it was to have I mean we include the Azure identity, SDK and that supports CLI credentials, um which it can.

A

So we can pull some some information about like default, subscriptions and stuff like that, where you would do it through a like an AZ account show command and that's what we were trying to do and it looked like there used to be a public API to be able to get that from the Azure core SDK and that was deprecated, so default subscription being a CLI construct, but not having a way to do it from the Azure SDK.

A

It was like you kind of needed to do it from the CLI, so it seemed like our choices were to take a dependency on the CLI or keep using this internal method to farm out to AZ account, show and I know. Kapil you found that that piece where it said.

A

Basically what that's doing is parsing the uh the Azure profile.json file in your home directory and it looked like that's what it did in like 90 plus percent of cases, but it seemed like it was also possible that that config information would get stored somewhere else like a Mac keychain or something like that. And so, if we tried to parse that file directly, there would be edge cases where it would fail. I.

C

Don't no, but that's the case because I'm on a Mac and it's very much like the credentials are in Keychain, but this is not parental information. This is like identity information that I, like preference, configuration stuff.

A

Then that may be, then it may be the case. I'd.

C

Rather avoid a depencing on the Azure CLI, because I don't like we don't install in our Docker images, like there's other reasons for why we shouldn't depend on the S on the CLI uh I guess in the context of like we would hit the metadata server so to speak and or we can be explicitly provided, but um yeah I'd rather avoid it uh dependency.

E

Like we don't generally.

C

Depend on clis um from the cloud providers if this seems like a weird case, it might be worth tracking through on the CLI likes to go. Stick a debug statement and walk up the chain from the entry point to see. If there's any other paths that it can go down. But um that's.

B

Fair, it's fairly find the.

C

Link permission is fairly because you.

F

C

Since we really encoded Json file- and that's that's really about it- um and if we can describe that directly, then I think we can save installations on uh or save some paypoints. Okay.

A

Yeah, that seems reasonably me, I. Think the where I tripped was that I am not as familiar with Azure and it looked like. There was a potential Edge case and I thought. Oh, let me just fix the most the simplest thing and then we'll come back to this. So if the next pass is to parse Json file that works for me um and again, no pressure but Mandy. If you do have any input on this, if any of this sounds like something you've bumped into uh inside, welcome yeah.

C

It's not as a matter of user percentage of like understanding, people don't bother to look at. But if the attacker welcome.

E

Yeah I'm not super familiar with Azure, either I just found that line and just got it working um for whenever I needed it, but.

A

Yeah yeah that worked.

A

Okay, so those were all.

E

A

So Kapil that was the yeah. That was the Azure piece. The other thing we see Dr builds uh Docker builds were failing, the only one that I know was failing with the C7 on left and that's right. This so.

C

They are open issues with c7n left um I I, don't know that I want to go down this VR route because it feels strange um sure, generally speaking, we um we build. We build the full image from entry, and the way this is set up is that it'll build the image from release artifacts as well, which doesn't allow for testing off a branch. Let's say um it did put in mind the question of why this wasn't failing in CI I.

C

Think it's gonna be only do the docker build for Maine uh NCI um as far as the seven unlocked image, the other issues that we actually need to get binary to be installed. We're currently using the C7 left images a little bit different because we're using chain guards uh Wolfie images, because it's all to have better supply chain security, minimal images, stuff, um I think in uh unfortunately, modifying those images requires adopting a bunch more custom stuff which isn't going to work in our Matrix, build on Docker images.

C

So I think at this point it's probably worth walking back from Wolfie to Alpine, um like I I. Do want to continue to make because he said on.

C

The left is typically deployed inside of application pipeline, so we want to try to keep the if people are using the documents we want to keep the dependencies set minimal, so it'll be different than standard custodian images which are typically Ubuntu based, but um I, don't know if that's going to be a huge that will that won't necessarily be a significant risk in this context, um because this is more of a you know: Source static analysis versus all the other capabilities on policies, um so this is I actually would like this.

C

This is a release blocker as well. We also don't have a corresponding GitHub issue for this, uh which I think we should do first. um Oh, that's, fair uh and then we'll label that, as early as Blocker in terms of getting a release out, uh I think there's still, but.

C

um I'm hopeful that we'll at least be practicing our release, automation, stuff for July. So we um that's it's more push button, but uh yeah this. This is definitely a blocker like because it prevented the less our signed images for the last release going out, um as well as blocking you out on the daily day of images.

A

C

A

Probably move this one to draft then and just leave it there for discussion and then open an issue or reference it and we'll take it. A different way sounds seems cool, okay, anything else on that or that's I guess that's what it is.

A

Okay and then select the that's the main agenda items then other other than that I think we're going to go into any yards or issues that people particularly want to discuss.

A

Ion I did want to talk about that uh that RDS won the multiple resources, but we can. We can come back to that because I've been yapping a bunch we'll see if anyone else has something yep.

F

D

um Yeah I mean I had a couple, um but it looks like um they have since been approved.

A

um The SCS ones are all taken care of yeah.

D

Yeah um one question: just from like um workflow point of view: um do you guys usually update through with like the git like tool.

D

um Because or like uh and like avoid, merge commits.

C

So you like gently speaking because we have people from all walks of life as far as their familiarity with Git. um We don't really require anything as far as um how like the actual process on the pr, we will do a squash merge.

C

um So anything we're also going to commit to the pr we'll effectively do a single merge as a single commit back into uh the history. I mean generally, the the request would be. You know if, um if you're getting reviews on things uh to leave separate commits after the reviews, just so that it's obvious, what's being addressed from review feedback.

F

D

B

A

Cool, so that was that was this one that already got merged and then this other one.

F

Yeah I took a look at this I had a question on this with regards to um capturing um email identities, but I think that's clarified now, I think it only captures yeah.

C

Good I I would mean my I mean if it's operator, email addresses, uh yeah less concerns just because we already pulled out with some tags anyways for delivery purposes. So we don't feel like we're done anything it's like in customer type stuff, then that would be a different concern and what I'm significant concerned.

F

Yep got it except that I think the sphere looks good and pretty much conflict is Resort now so potentially we should be good.

A

Any others issues or PR's people want to flag for discussion.

B

um Yeah I'm not sure this is the right form, just be so 8620, so the initial CI provider commit um so just a question out to anyone here. Just if there's anything uh left to do, we don't have anything handing on our side that we know of so.

C

I think the biggest like I think the the last known thing I think got addressed. The problem is the pr is really big, so it takes up and.

B

C

Of course, this is just a question of like going through the whole thing from start to finish, but I'm not aware, I'm, also not aware of any outstanding at the moment so, but it still requires you could jump through the whole thing again.

B

C

Robert E froze.

A

I think we said: okay, we lost Robert I was until I saw you talk, I thought, I dropped.

B

You guys, did you hear me.

C

B

I just uh saying, if there's anything, you notice, that's missing um that you know maybe isn't urgent, but we we can phase that out and just figure out when.

C

To put it in definitely trying to.

B

Prioritize getting.

C

Getting things in so any review comments would be on. What's what's going on right now, um when will your future enhancements out PR sure sure.

F

Okay, yeah, we don't have anything. I was trying to see if um you could get an eye on the SNS too long message: error, which is PR 861. It I think it's still in craft mode, but um just wanted to see. If that's what a couple you were looking for with regards to getting this issue resolved.

F

Whenever you get a chance.

C

It will probably be this week, but I could not tell you when um sorry SMS, yeah.

C

A

Okay, so this is just trying to customize that okay.

F

Like dynamically batching resources for snss keywords transporting.

C

Yes, no I think it's doing the right doing the thing I want to structure differently because we have to deal with compression, but um uh yeah I'll try to do some feedback.

B

C

B

A

So your concerned, there Kapil is that we we just need to make sure that we're checking. Oh no, so we are encoding. Oh no, we're packing.

C

No, this is good double uncut things. This isn't quite right.

C

Yeah, this isn't actually it. This will end up double encoding messages, um the antennas that we actually assembled the envelope manually at the end too, so we don't get because right now this is serializing the message: there's still housing, each individual resource adding to a batch and then saying Json serial is the batch, which means you get a double encoding on the resources. So it just isn't quite right.

C

A

I did want to point out this one. This 8614 I think that this came through as a fix to the propagate tags, uh National, Property or copy copy related text for RDS and redshift, but it was we made. You ended up making the change at a at a higher level here in in queries, I think we were it's just worth calling out in case. Anyone comes across. This may fix other things, which means there's always an outside chance that it will break other things.

A

um What we were doing here was that this piece, this code path, comes up when we're firing an event-based policy or with certain other spots like like in copy related tag um and it when it tries to look up resources uh given a list of IDs, and we were dying on resource types that had a scalar filter type.

A

If we had more than one identity and so fixing this fixed copy related tag, it could also fix some uh some event-based policy triggers, but uh mostly just want to call that out just in case I would hope. The only other side effect is that some other things start working.

F

Yeah we haven't like really pulled it into our uh platform or our environment, yet but I think once we do that I'll definitely have more visibility in terms of something that this might end up. Breaking.

F

But yeah thanks for the help on this yeah.

A

Well, so yeah thanks for a couple of all the testing I know is it's a little tedious with those those data sources getting them up.

A

A

I think that it, unless there's any follow-up discussion or other issues or PRS, that folks want to talk about.

D

C

Is there are black um Ken took up the mailer formatting, so I think that's probably going to go through separately. Then the fermenting the rest of the tree, um because I think he's got some active work. He wants to do on the mailer and he wanted to take care of that first. So as.

E

Soon, as this passes.

C

See I I'm going to just do this one by itself and rebate the other one uh once this is landed.

A

Yeah I started looking at this actually I'm glad you mentioned this one, because I did have a couple questions about it. So one is that he was using the um the 100 Lang length here. I think we were depending on the Black Version. We were either implicitly using 88 or 100.

C

We could figure I, think I thought we were yeah. We were in my prior project, comma.

A

um Yeah and that's what it's not configured there it's configured for rough, but not for black, so yeah I was gonna. Let him know that we should add that configuration there and add mailer to the format set uh as part of this PR. Also, if there's gonna, if we're gonna, combine a black with a refactor I'm wondering if we should split it out and then add the format pass to like an excluded commit list. So we can filter those out later or if we, if we care that much.

C

Related to the the mouse reformat I'm, okay like leaving it out um suppose we get like a you know. The rinky dig one-liner commit I mean which is fine but like it feels it feels part and parcel, like otherwise, I think if we were to structure it, otherwise I guess doesn't matter if it comes before or after, but I mean I. Think it's it's pretty reasonable to I understand what you're saying you're like a semantic change here, um but it's a one-liner so I'm! Okay with that.

A

Yeah I'm fine with it and just I, know that, as we start doing, and this ties in a little bit to the uh the oci commit too, because I think there was, there was a different length there and I was thinking. Well, if we're gonna, if we're gonna, have a formatting, we should just keep that the same the same line length and then add oci to the format specs so that that lint pass touches, touch that out of the gate.

C

That's a good point. um We can probably directly like push some of that stuff to the PR's that are extent um to make it a little bit easier but uh yeah. Those are all good callouts.

A

Okay, oh well, follow up on this one then, uh but yeah thanks for bringing that up because I forgot about it.

A

Okay, so I think that's it for this meeting, then uh thanks everybody for coming uh more questions, any other follow-up stuff that comes up after we can handle in slack and uh oh Kapil. You also pointed out. We should have these notes going in as GitHub discussions. So if we have any direct meeting follow-up, we can have it right in GitHub um and I had not been posting those there. So I'm going to start doing that with today's actually.

C

Do we have, uh are we do we have the flow for meeting recordings going into our YouTube channel? Yes,.

A

We're putting them in it's a bit of a manual thing actually, unless somebody knows of a way to do it uh directly from the Google me to YouTube, um but we've been putting those over there and then because they get added to the community meeting playlist. They also end up in that uh they get transcribed and they end up in that list at uh where is this spot?

A

Oh yeah, that's meetings.io!

A

So a little bit of time after they they go through. So we see that the last one here you'll see Links of the meetings and then transcriptions here. If folks want to review stuff.

C

So the other thing that came up in that context was if I go to the Community page on the website.

C

Advice, I, don't know if there's a way to get a feed on that, um because currently strong cleaning for like two or three years ago, and like just wondering about updating our page to like Point people to the link, as opposed to a multi-year-old meeting.

A

Yeah yeah the playlist, so so once yeah we add them to the playlist, so they go there. They're just yeah.

C

Okay, that works it's just so that that one that the the image one the.

A

C

A

Know a while ago, yeah gotcha I wonder if we can edit the playlist I, just reverse sort.

C

A

It's in case it's not clear, um I'm, still very much figuring this stuff out um arm wheel, so yeah, uh but that's a good call out too I'll. Try to get that updated, that'll, be cool.

A

All right, oh thanks, Mandy yeah I had looked I was like. Oh, you used to be able to directly move things, and now you can't download it and re-upload it. So maybe there's.

E

Like it's, it's live stream, so you can set up the.

A

E

And it'll it automatically saves live streams.

A

Yeah I saw something about live streams and I was hesitant to do that because occasionally, we'll have a discussion on someone asked a question and they post credits or account IDs sensitive info and it's nice to have that editing a while, but I like the option of having the edit pass to pull that stuff out. Maybe we don't need it anymore, though. Maybe we're in a live streaming world and we should just live stream. The thing do we is since we're we're in a community meeting.

A

Does anyone have opinions one way or the other about live streaming as opposed to uh to just taking the recording.

E

A

Does anyone opposed to live streaming, whichever.

E

A

B

Yeah, whatever makes your life easier, I think.

A

Yeah all right, if there's less manual, steps and we're okay with that I'll just try to make it really explicit call it out before we start it or at the beginning, so that then it's just really obvious to folks wow.

E

You can always go back in YouTube and blur out images too.

A

Oh cool thanks, I genuinely didn't know that yeah.

E

A

All right cool! Well, thanks now we have something to look uh look forward to for next time. I'll have to look into live streaming, foreign.

A

Thanks everybody for joining for the contributions and we'll see you in Slack.

F

B

You thank you goodbye.