GitLab Customer Success Skills Exchange, 20 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Agent for Kubernetes & CI/CD tunnel w/ generic impersonation 20220420

Description

Walk through the latest changes to GitLab Agent for Kubernetes and CI/CD tunnel. Learn how to the various impersonation options can be used from Cesar Saavedra

A

B

A

Welcome everyone to another skills exchange session. We have uh cesar here today to talk through, maybe some hands-on activities. If not, you know we'll do some demos as well, but cesar I'll go ahead and let you kick us off.

C

Hello, everyone, thank you. So much for joining. uh You should be seeing a screen uh that says cs: skills, exchange.

C

Great so as I was mentioning for those of you watching the recording uh the there was a prerequisite of creating an mr in this project before this moment and unfortunately, no one created dmr.

C

So what I'm going to do is I'm going to go ahead and just demo, this lab to you and you can go ahead and create the emr if you want and what I'll do is after you create the admirer, I have to give you a special access to or access to the to my cluster and then I leave the cluster running for about a week, and then you can do the exercises yourself uh on your own time.

C

So let's get started so um I have this. I just have a couple of slides so today we're going to cover the gitlab agent for kubernetes or g84k, and also the gitlab cicd workflow for kubernetes, also known as the cicd tunnel. That was the old name and we're going to cover the generic impersonation using the tunnel.

C

This is me: if you have any questions, please feel free to reach out to me uh via my handle or slack or my email.

C

All right, so what are you gonna learn from this lab you're gonna learn how to configure your own ga4k on an existing kubernetes cluster you're, going to learn how to use the pool based githubs to create your own namespace.

C

If you attended this, the sko lab the first five minutes will be that and then we're going to do some extra stuff after that. So this is a. It has a lot more content than the sko lab that you did also you're going to learn how to use the pool based git ops, to create an instance of nginx in the cluster.

C

You can learn how to configure the ci cd workflow or the csd tunnel and we're going to be accessing or we're going to be through the tunnel. We're going to be impersonating the agent, which is the default, a ci job and also a service account in the cluster.

C

I've provided a link here and actually what I should do is also I'm going to do this. I need to share this with you.

C

Copy link and then I'm going to go here and chat.

C

There we go. I just shared the link in chat to the to these slides, uh I'm clicking on the wrong thing now. So in this slide, you see there's a link here. This technical video resource shows you how to set up uh the class. A cluster gke, specifically with the ga4k and deploying the gitlab, manage application applications uh to the cluster using the cluster management project.

C

Template specifically in the in the video you'll, see that I'm deploying prometheus cert manager and the ingress and then lastly, this link here takes you to the instructions of the readme that you're about to see.

C

This is a very high level, logical view of the two approaches that we do for get ups and supported by the agent.

C

So on the right side, this is the pool base, get ops, and this is where the agent actually pulls updates from git, as they happen from gitlab as they happen, and then updates the infrastructure and on the left side.

C

This is basically it leverages the cicd tunnel or the csd workflow for kubernetes, and this is where you when, where you're accessing the cluster from your ci cd pipeline- and this is also supported by the agent, the specifically talking about the ci cd workflow or the tunnel, if you can see the difference between the the left picture here- and this picture here- is that in this case uh we are just on the right side. We're showing uh the support is just for kubernetes yeah.

C

Let's see, and as I mentioned before, we're going to be doing a lab where we set this up and then we're going to be impersonating.

C

You know the agent, which is the default cr job on a specific user in the system.

C

All right, so, let's go to the demo.

C

Is there a question already?

C

Is this setup process going to change uh actually uh a funny story? Mirko uh yesterday I um I I asked my teammate to be my guinea pig to test the readme file. I mean the instructions here in preparation of today's uh webinar with you guys uh and we noticed some changes in the product already, so I had to quickly update the instructions.

C

So these instructions follow the the setup of gitlab ourselves yesterday and what was changed really we'll get to it is that when you install the agent before we were uh you needed docker and now you don't need docker running on your laptop. You just need to have helm, because that was the main. There were some other minor changes in the ui, but they were minor.

C

So just in case, uh please feel free to ask any questions at any point. Okay and- uh and if I miss the chat, just speak up- and let me know if I can answer your questions because I'm going to be busy following these instructions all right, so, let's um so the prerequisites for for running. This is basically three things you need to have since we're using gke.

C

You need to have the g-cloud cli installed on your laptop, you need to have helm and you need to have cube ctl.

C

Also, the prerequisite was the creation of nmr uh that is basically the create that will create a namespace, a personalized namespace for you. This was similar to the lab that we did in sko, and this is the part that I need to do once you create the mr and I notice uh you know I notice who's created emr then I need to add you as a kubernetes engine admin to gcp and also give you um maintainer access to the project.

C

You know you have superpowers, so please be careful, don't don't, kill my cluster, I'm going to leave it up and running for a week for you to try these uh these instructions out all right. So let's continue. I'm just gonna. Do the labs these labs myself and you can uh they'll get recorded and you can watch it later and try yourself later so. The first thing is, I have an open terminal which and I've already connected um connected to the cluster. Now let me share.

C

Let me share what this will be for you. I think it's right here, so this is the cluster name. Css skills going to copy this.

C

So after this call for those of you that have created an mr indicating to me that you want to try this lab, you will need to execute the following.

C

And put it in the chat, the gcloud uh command here that I just pasted. Just as it is you copy and paste it in your terminal window and you will have access to the cluster from your terminal window.

C

And don't change anything in this command, please just paste it paste what I've um put in the uh in the chat right now just to the way it is all right. So then um you can double check once you paste that that you have access to the cluster, which I already do, I'm all set up here. Okay, so I'll skip these steps- and here I just put out if you have any kind of k9 tools, you know you can, you know, observe the cluster. You know browse through it.

C

If you want, uh I I don't use any. uh I don't use the k9 as tools I just uh you'll see, but just use simple cube. Ctl commands all right, so next um section is about create a merge request for configuring, your own agent.

C

So at this point you will have created an mr that you have not merged yet that will add your own personal, personalized namespace, so that mr we have not merged yet, but it's existing.

C

In fact, I need to do it myself.

C

So let's go here. You will click here in this issue which takes you here and then this gives you the instructions, and it also gives you a one-minute video on how to do it. So I'm just going to go through this, so I need to go here and I need to go into web ide.

C

And I'm gonna go here and I'm gonna say create directory and, let's just say my handle is john doe jdo.

C

And then under there, I'm going to create a file and I'm going to call that file jdo ns dot, yamo.

C

And then what I'm going to do is I'm going to copy this example.

C

And I'm going to change here, the c sub editor handle for jdo handle okay and leave the rest the same now here we're going to go ahead and create a new branch, but we're going to start a new merch request and in this commit message, let's.

C

Let's put um adding joando namespace.

C

Now we're going to start a new merch request in its own branch.

C

You can assign it to yourself and then create.

C

Now, at this point, we're not going to merge this, mr we're just going to leave it in the back burner, we're going to merge it later in the instructions.

C

So, let's continue so I'm done with that prerequisite.

C

So we're here right all right, so let's see we're going to go ahead and we're going to create a second, mr and this second, mr, is going to add the agent configuration okay. So let's open we're going to do this project.

C

Let's copy this right here.

C

And in here we're going to create a new, a new, mr and we're going to create this config.yml we're going to use the web ide so that when you use the web id, the ui allows you to create. It creates a branch for you and a brand new, mr, so it just makes it uh easier. So you don't have to if you, if you just do an edit you'll, have to find the branch and it's just it, takes more steps.

C

So we're gonna go like this. We're gonna, say: web ide.

C

Right and then.

C

We're going to create a new file.

C

And here we're going to replace the handle for jdo.

C

B

C

And then we're going to go ahead and copy.

C

This text here and we're going to paste it into that config.yml.

C

And we're going to replace here your handle when you do the lab you'd use your handle. Okay, I'm going to use jdo here and I believe that's all. There is.

C

Yeah then we go ahead and commit we're. Gonna use these words in the commit message.

C

All right so we're going to go ahead and commit.

C

Agent configure we're going to create a new branch and we're going to start a new merch request.

C

We're going to assign it to myself and your case, you send it to yourself, create the merch request.

C

C

So now we need to go ahead and merge the this second, mr because this by merging it we're going to add the configuration for the agent for john doe. So let's go ahead and merge this.

C

And while this merges the next thing, we're going to do is we're going to go ahead and register.

C

The agent with gitlab.

C

So, let's go back to ga4k, we'll use this window here.

C

And then infrastructure, kubernetes clusters.

C

We're gonna click on connect to cluster.

C

And in here we should see the john doe one and we're going to go ahead and say register now. This has changed from the last from last month. This is not no longer docker run, as you can tell. This is helm commands so we're gonna copy these.

C

You will get your own token when you do this and let's.

C

I'm gonna save this here.

C

I need to move you guys from my other screen, so I can paste that.

C

So when I save this on a local file.

C

I'm going to close it.

C

And then, what I'm going to do is I'm going to just go ahead and show you.

C

So this is what I did okay, so I saved it here. So what I, what I'm gonna do next, which is in the instructions also you're gonna, add you're gonna, add your handle here in the install parameter and with a dash, and this will personalize the pot name.

C

So we come here to the install and we're going to say john doe dash, okay and then we're going to copy this.

C

This out of the way.

C

And we're going to paste it here.

C

C

ah Wrong command there we go all right and now there's an agent running. It's been running for nine seconds, so the running those helm commands has deployed this agent called john doe get lab agent. In your case, you will see the pot name. Have you will have your handle here in front and you know every one of you will have your own agent running in the cluster.

C

So let's go back.

C

So we're done with the agent, so one thing that we can do now is let's- um and it's right here in instructions: let's go ahead and do a tale of the log file of your agent.

C

So let's copy this command here and let's do it here on top we're going to paste and there we're going to change we're going to use here, the pot name for your specific agent, and in this case the pot name is right here.

C

Okay, now this is this is gonna. It's gonna. Do a watch right. It's you're gonna see us changes as the agent reacts to changes. You're gonna see output here, so we'll leave that screen the way. It is all right. So now now we're going to now that the agent up is up and running and let's go ahead and uh merge the namespace.

C

Mr this one here now before we do that. Let me show you.

C

The configuration.

C

That needs to go away there, so this is basically saying the configuration of the agent is saying um that this is the directory that is going to be observed for changes, and if there is any file in there with an extension, yaml, yml or json, then the agent will react to those changes.

C

So we're going to do is remember the mr for the namespace is going to drop basically a yaml file there. Let me go back remember this is gonna drop this new file under manifest john doe, which is the directory that the agent is listening or is observing for changes now before we do that, let's make sure that.

C

Let me look right here, so these are the name spaces right now in the cluster notice that there is no john doe.

C

So let's merge and let's observe the agent should detect a new file and create the namespace after a couple of seconds.

D

C

So, let's see if the namespace got created.

C

There so john doe namespace has been created.

C

It's there, so now the next step. Now that everything is set up.

C

C

A git ops flow, let's, uh let's exercise to get up slow, so I'll get a flow is basically includes the creation of the issue, the mr the the commit and the merge, and then we watch the agent um basically update the infrastructure with with that new new merge, so to do that, let's go ahead and create an issue under ga4k and open another window. Here, let's move it here.

C

Let's create an issue here, new issue and we'll just call it here. I think we'll just say: provision and nginx server.

C

And then I can assign it to myself we'll skip the description, we're going to create the issue.

C

The idea is we're going to go ahead and create the emr and we're going to launch the web ide.

C

There we go now. The idea is in in dmr all the collaboration happens among all the stakeholders. Working on that on resolving that issue in this case is provisioning, a single instance of nginx, so from inside the web id we're going to navigate to. In this case, it's going to be manifest. John doe, in your case it'll, be your handle, we're going to create a new file there, nginx yaml.

C

And then we're gonna copy and paste.

C

C

And we're going to replace this.

C

With the namespace of john doe,.

C

Space, let me just double check that this spelling is correct: j, don't name space.

C

And uh we go ahead and we're gonna basically create the merge.

C

Go back to the merge request and as soon as it's ready to merge, we'll merge it.

C

Now remember that the agent is listening or observing that directory for john doe, so this is basically adding a new file, a new yaml file and it's going to drop it in that directory.

C

And then, when that happens, the agent should react.

C

Let me see, uh I may not be fast enough to show you the before and after yeah it was too fast, so the engine x was already created by the agent. The agent saw that new file and it instantiated a new in a new pod. That is.

C

Nginx, okay, so that's the first part of the lab any questions before we start move on to the cicd tunnel.

C

You can drop your question in the chat or just.

C

Ask through live through your mic: no okay, just interrupt.

A

Me document either.

C

Oh there's a document. Sorry, I wasn't following the document all right. No.

A

No you're, okay, there's there's no new questions.

C

Okay, good good, so please interrupt me if you have any questions: okay, all right! So now, let's move on to the ci cd tunnel. Now the cicd tunnel leverages the g4k the the agent to access to securely access, the the kubernetes cluster okay. So now we're gonna switch topics and we're going to be doing what we just did was uh this right here.

A

Sorry cesar looks like we have a question from scott in chat um scott. I don't know if you want to.

C

Come up yes, yes, the answer is yes uh the the way that uh we've designed the agent, the uh you probably want to want to have an agent, maybe per project. It just depends on the needs of the organization or the department how they want to split the agents but yeah you want to have you know. The expectation is that you will probably have multiple agents and the classification depends on uh on the organization. It could be by department by project et cetera, but yes, all right.

C

So we just covered this pool based git, ops and now we're going to be doing this, we're going to be we're going to have a pipeline that is going to be accessing securely accessing the cluster okay and then you can put whatever you want in the cluster.

C

You know you can act basically act on the uh on the cluster and do things to it and and get information from it and create things, and uh you know kubernetes, artifacts and elements. So um let's go back.

C

And this part is we're going to be accessing the cluster through the pipeline impersonating, the default, which is the agent think of it, as you know, has all the power and then the ci job and a specific username.

C

So let's go ahead and we're going to go to this group, I'm going to and we're going to create an application called called sample application.

C

Just open a new window, ah what did I do?

C

Oh lordy, let me see.

C

There, okay, now this group has a project which contains the configuration of the agent. We call this project ga4k. If you click on that you'll see.

C

This has all the information about the agents and where we've been dropping, we've been dropping the nginx in there and all that, so we're going to go ahead and create a project at this level and then we're going to tell the agent uh we're going to give the agent is going to give permissions to this project to to basically securely access the kubernetes connection that the agent is maintaining so new project.

C

Bland project paste and then you can just make it public and let's leave a readme there.

C

Okay, so the brand new project is there, so we're gonna go to uh okay. Now we have to go to this folder. We have to now update the agent configuration okay. So we're going to go to this folder.

C

And we're going to navigate to gitlab agents jdo and we're going to open config yaml and we're going to be replacing the content with this now um good luck, agents, john doe!

C

Now we could append. We could append uh all these uh parameters to this file. If you like, okay, so you can do, uh let's just do an edit. In this case we can just merge it to uh yeah. Let's just do an edit in this case instead of web ide.

C

We'll skip the creation of the issue on the mr.

C

And uh the instructions say uh just uh I'm sorry, maybe I made a mistake. um Actually we need to append I apologize. We need to append. I made a mistake, so let's append this here like that, so let's append that and then we just commit to the main branch.

C

And you probably will see the agent should react.

C

There it just reacted all right, so it updated its configuration all right. So the next thing is and we're going to create a pipeline in the new project that we created in sample application project.

C

Okay, so let's go to a sample application project and we're going to do a new file and then we're going to create a gitlab ciamo.

C

And I believe we were here.

C

Yes, a new project, we're gonna, say new file, we're going to say gitlab, like that, there we go, and now we need to replace this portion here with the name of the agent and the name of the agent is john doe. I believe I remember correctly.

C

If you want to double check, um we can go here.

C

And then I believe it should be listed here.

C

Yeah, the name of the agent is jdo.

C

um We were here right yeah, so we update the cube context to this and I think that's all we need to do and then we commit to the main branch.

C

All right, so this has added a pipeline to this project.

C

I believe we can go ahead and, let's see if it started running already.

C

It's right here right, yeah yeah, it already ran it's running. Look the agent already reacted. um So let's go to pipelines for this project. Oh and it passed okay. So let's go!

C

It just ran uh a few seconds ago triggered 30 seconds seven seconds ago. Let's go into the only job in there and stage and remember the pipeline basically had to recheck the pipeline.

C

It had two commands to be executed, get context and get parts all name spaces and, as you can see they both these commands both executed successfully and we were able to access the cluster successfully. Okay,.

C

And here you can see what we saw earlier from the terminal window good. So now we have, the default is impersonating the agent, and this would you. This is what you just saw the the pipeline when it ran. It ran with a default impersonation, which is the agent which has uh access to the cluster, um with no restrictions.

C

All right so now, let's move on to the next.

C

We basically just did this okay, now, let's impersonate the ci job, so for the ci job. Let's go back to the configuramo this one here for the configuration of the agent we're going to append this right here to the bottom of the file.

C

So, let's edit.

C

And, let's add it there now, the spacing in front matters, okay, so make sure that when you copy this, you you're copying all the way from the beginning of this line. The line the blank spaces here are there on purpose. So this is going to make it's going to impersonate. Now the ci job.

C

So let's commit- and I believe this is going to run the pipe another pipeline.

C

Not there sorry, actually, no, it's not going to run the pipeline because it's in this there were no changes to sample applications. So we need to go to pipelines here and now, let's rerun it and now remember this time, the pipeline is going to impersonate the ci job.

C

Here the agent just updated his configuration to access as ci job. So let's run the pipeline.

C

Now the pipeline here is going to, one command will succeed and the second one will fail.

C

And the reason is, um when you run when you impersonate the ci job, uh there is um a username that is basically created for you for that single execution. In this case the user is called ci job and then the number of the job- okay.

C

So and it fails because this user doesn't have the the privilege or the permission in the cluster to be able to list the pods in the cluster and that's why it fails notice that the first command it runs fine, because it doesn't require special permissions, and this you know, you're able to see the context which takes us to the third case.

C

How do you impersonate a specific user or a service name inside the cluster? And that is the third case which is right here.

C

So now, let's assume that we want to impress this could be jane. I could you know I'm assuming that it's going to be a service account called jane, but you could do the same thing with the ci job here: okay, so in order to be able to impersonate a service name or a user, we need to create a service account for that user. So let's go ahead and create jane.

C

In the cluster there, then we have to create a cluster role, we're going to name that cluster role get pods role, which is basically a role that will allow you to list and watch and get all the pods in the cluster.

C

Then we have to create a we need to bind it to jane. We need to bind that role and we're going to call the binding. Allow get part role.

C

I'm going to give that permission to jane.

C

Now now this is for jane right. So now we also need to give the agent the permission to impersonate another service account that doesn't come out of the box. So to do that, we need to run this command so that we're going to create a cluster role called impersonator that allows um to imper. It's basically allows an account to impersonate another service account and then we're going to assign we're going to bind that to the.

C

C

Now here I need to update.

C

Remember the agent is jdo dash kid lab dash agent.

C

C

All right, so the last thing we need to do is we need to go to again to the configuration of the agent and we're going to replace the axis as for with this code segment.

C

So this is here we're going to edit the configuration of the agent and we're going to replace these two lines.

C

With that- and this is what it's saying is when you run this pipeline- I want you to impersonate jane, I'm gonna commit.

C

And now let's go and run the pipeline again, but this time the pipeline is going to impersonate jane, let's go to pipelines.

C

And then then, let's run the pipeline and the agent just reacted and it updated its configuration to as impersonate jane gonna run. The pipeline.

C

And let's go here.

C

Now we all those commands that you saw earlier, gave jane the ability to list parts and also gave the agent the ability to impersonate jane. So, as you can see, because of these commands that we executed here for the right to give the different components. The right permissions, jane, jane, has been impersonated here and jane is able to list the pods that are running in the cluster. If we hadn't done the created these permissions or cluster roles and bindings, the second command would have failed, just like in the ci job case.

C

And that's all I had that concludes the demo.

C

uh Does anyone have any questions.

C

I hope I hope this was useful to you all.

E

I do actually okay go ahead, um so can the the configuration of the user also depend on the user which executes a pipeline.

C

When you use impersonate it'll depend on the uh user, that is being impersonated.

E

C

E

Other way ask what other access options do we have here? Oh okay,.

C

So those are the three that we have so far. We have the default, which is basically you're impersonating the agent um pretty much gives you full access and then you have the ci job, and then you have impersonating service, name or username. I believe, let's see gitlab. This is ci cd time all right gitlab um they changed the name.

C

Agent for kubernetes- and I believe, if you go here see I see the workflow.

C

Here, uh authorized environment certificate, impersonation, restrict project and group access.

C

So in this case I chose group with the idea that um any projects that I create under a group will, you know, will have access to the agents. But you could, uh you know, do select a specific project if you want for the impersonation step.

C

So that's the agent, uh the cicd job that accesses the cluster, and this tells you what the names you know the usernames will look like. Remember. We saw that lexia job and then a number like this one and then the third one is a static identity. In this case we use jane, and those are the three options that you have we have so we provide so far.

E

Okay, so this basically means if I forget, to configure uh access s and just have the project configured um everybody who has access to that project. Has agent access right right.

C

That's the default. Yes,.

E

C

Yeah yeah any other questions.

D

Hi cesar adriano here.

C

First, thanks for the presentation, oh you're welcome,.

D

Just a follow-up question on what was asked before you mentioned that uh the agent is intended to have uh organizations using multiple agents in a same.

B

D

But effort after you show us, the impersonation sounds a little bit redundant to me. We could use just one agent and use impersonation and we would have different service accounts, deploying different research in the kubernetes cluster, and this would facilitate audit and tracking of this operation. So how do you see this? uh If you are using impersonate, you still feel that is necessary to have uh multiple agents in the same cluster.

C

So um from from customer interviews, what I've read is that they like the customers, would like to split the access.

C

um For example, it could be by department so that there's a need to do that, but other customers also want to either even have a more granular restriction than that than you know, having everyone, an agent per department, for example, and having everyone in a department access a cluster through that single agent uh and in those cases you may want to restrict access to a single, a single specific service service name.

C

It's just a matter of um you know I want to say. I know this sounds a little. um I guess it's not a direct answer to your question, but it depends on what the customer needs right and, and the good thing is that we provide both. You know they can use agent with uh an agent per team or per department and then leave everyone in that department just access to that cluster with the default agent impersonation user default.

C

But if you want a more granular restriction, maybe you have. I don't know you have a a testing team or a group of people that want to do the testing and you want them only to do the test you may want to just maybe impersonate a specific service name that they are allowed to use.

C

So it just depends yeah.

D

Yeah, I can see what you're saying, maybe one agent by department and then using person eight, maybe to impersonate a specific uh developer from the part uh the department yeah. So.

C

Yeah yeah and it's up to to the organization right. They may not need that, but we provide it for specific use cases.

D

All right, thank you. So much you're welcome.

C

Any other questions.

C

So so you have access to this um to the project. Please, if you want to, if you have any ideas for improving this, let me know you can, even you know, do a merch request if you want to add something or delete something to these instructions.

C

Let me know I also have this whole set of.

C

Whole set of uh exercises that I've been trying to keep up to date. uh This one here.

C

So this is there's a bunch of cd labs, um you know for rollbacks, feature flags etc, and this one get ups with gitlab, the very bottom.

C

So the first part of this um lab uh covers the a lot of the integration that we have with terraform and then, which is what we call the push based uh and terraform is you know you can use terraform from non-kubernetes uh infrastructure targets right uh and, and then the second part covers the, um which is this is very long. Actually.

C

um The second part includes some of the content that you've seen today, which I really have to update, because I just discovered yesterday that some of the uh some of the ui has changed so as soon as we're done here. I'm going to update these these instructions.

C

uh The second part of this one is uh using the agent, basically okay, which is similar to what you saw today, but with screenshots, which, unfortunately, unfortunately, they the ui changes a lot with gitlab. So um I try to keep up with the screenshots, but I'm thinking that I mean have to do without this update actually do without any screenshots, because the ui just changes so much that you know it just requires a lot of time to update a screenshot, because I have so many, but this is another resource that you have at your disposal.

C

um If you want to try out- or you know, um you know, learn more about the different or use something like this with a customer to create a a demo.

C

So that's uh that's right here!

C

I'm gonna share this with you in the chat there. You go cool, so that's that's all I had for you all today.

C

So um any other questions.

C

Okay, so pallavi I'll pass it back to you.

A

Wonderful, thank you so much for your time. um Yeah, I don't see anything additional in the dock, uh but if you have any questions feel free to add them on there. This recording will be ready by the probably by the end of the day and I'll post it on customer success, just as I always do, um but thank you so much for your time today. Cesar. Thank you for the great demo and thank you for the great questions.

A

We'll see you all next time.

C

Thank you all for attending. Thank.

A

C