Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 3 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing the Superpowers: Who Loaded That EBPF Program? - John Fastabend & Natalia Reka Ivanko

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Securing the Superpowers: Who Loaded That EBPF Program? - John Fastabend & Natalia Reka Ivanko, Isovalent

eBPF has become an increasingly popular technology to build all sort of tools from networking CNIs to security tools. eBPF has the ability to inspect nearly any kernel data structure and modify networking packets and even user space data in some configurations. It has recently become cross platform with a Windows run-time and is now widely available on most Linux distributions and cloud platforms. It even has users at Blackhat (BlackHat USA 2021: With Friends Like eBPF, Who Needs Enemies?) and Defcon creating potential malicious uses for eBPF. Precisely because it is so powerful it is incredibly useful, but it raises the question who is watching eBPF. The Linux kernel community has been building a solution to securely monitor and enforce who can load eBPF programs and what kind of programs are allowed to be loaded on any given system. In this talk we discuss a design for eBPF auditing and security and use Tetragon's (an open source eBPF based security tool) to show an implementation. This will show security teams how to restrict what gets loaded on a Linux system and who is allowed to use it. As well as how to create an audit log and time series database so we can go back in time to discover the who did what, when type of questions that can not be answered today.

A

um So hello welcome everyone on one of the last talk today, so it's going to be about securing the superpowers and who loaded the evpf program, so basically auditing, ebpf programs, so who is speaking today? I'm Natalia I'm, a security product lead at ISO villain and here is with me John, who is a tetragon lead and psyllium maintainer, as well as staff engineer at ISO Island.

A

So I wanted to start with a background and the motivation to this talk. First and, as you already know like evpf, is on the rise or already Rose. It became one of the leading Linux kernel technology today.

A

So more and more projects are using it under the hood for different use cases, for example, networking observability and security. So this is just a small subset of projects that are using ebpf, but this would give you a glimpse of what can be done in different areas.

A

So, for example, in networking it can be used for high performance and load balancing in other data centers, as well as in Cloud native environments. So one good example is catron, which is a high performance load balancing from Facebook that they created to replace ipvs. This is another software-based load, branching solution, so they switched actually to ebpf and saw a massive performance increase. This is actually open source, so you can actually go to the GitHub repository and check it out. It's Facebook, specific, but good for other Linux based software infrastructure systems as well.

A

So we have celium, which is providing networking load, balancing security for kubernetes services on the observability side in the middle we have BCC and BPF trees, so these are for application, profiling and tracing so, for example, understanding what my application is doing. What my my application is not behaving the way we expect, for example, how many block I occurs it's using and so on.

A

We have Hubble, which is the visibility component of psyllium, and it can use, for example, for Network policy, troubleshooting auditing, so, for example, which network flows are denied by which network we see what is the source spot and so on, and then for security. So we will deep dive into these use cases later, but we have tetragon and Falco, which is applying BPF for container runtime security, so inspect the kernel function, system cause and figuring out.

A

If a malicious behavior has happened, we can also Implement, for example, these privilege policies, and then we can create proflies based on the observed event, and then we should say like these are only the allow Demons by that Source. But and then we should prevent everything else. We can also use BPF for preventative security, so terminate any kernel, functions or system calls inside the kernel instead of having like a user space agent, observing it.

A

So, for example, these are two security use cases that we can do. One is data ex filtration, so a security team can observe occupancy's, namespace and figure out like which, which parts were the most outbound to occurs in the last hour? Is it expected who sent out the most traffic? Can it be detox filtration, and we can also do like file Integrity monitoring so which pods or workloads open, sensitive files with which binaries who was root? And is it even expected?

A

We can also do capability and namespace success monitoring, so security team camera, for example, certain name spaces and ask questions like which ports were running with cops's admin or cut net rule. Do we expect this which which ports had, for example, whose speed or network names with success? Do we even expect this? How long who started this spot, and so on so EVPs became cross-platform and windows machines with the windows runtime? Recently it's available on most Linux distributions and then all the major Cloud providers also supports.

A

So um as a motivation, since EVP has become so powerful security teams needs to answer functions like who is watching evpf um during a secure, it's actually important to keep track and audit which BPF programs have alluded and then which ppf's map were created.

A

So what does? How did it mean exactly like who? What did it, which guberty's workload which process of each binary from which ancestors? When was it loaded? um Should this program be expected, have you seen this program or process before and should this process touch BPS at all? So these are the questions that we are trying to answer today with um tetragon.

B

Okay, hey everyone.

A

B

All right so first I want to talk about tetragon, so that's our tool, right and- um and we saw some of the use cases that it solves, and we saw some of the dashboards that you can create and so I'm going to kind of talk a little bit about the architecture here from this um this cartoon, but at least it gives you the big kind of bullet points and then from there we can dive into how we address the BPF monitoring piece of it that specific part that the talk is going to focus on so first, what is tetragon it's our security, observability and runtime enforcement.

B

What this means is you take BPF, you hook the Linux kernel and you can hook all of these locations right use it. The tcpip stack, you can hook system calls to process execution file systems, you get all of your c groups and namespace monitoring, right and and what tetragon can do inside is kubernetes since we're at the cncf here is.

B

You can take all of that kernel level data and put on top of it the kubernetes metadata and get something that operators and people that monitor and manage kubernetes environments can make some sense of right, because if you didn't do this- and you just said- okay I'm going to put a BPF probe here- maybe I'm running BPF, trace or I'm running BCC tools, if you're familiar with that, these are kind of low level.

B

Bpf things what you would really just get out of your system, then, is uh you know, PID name, a random PID launch some file fd3 and oh by the way, it's in some random C group on some random node in your system right. So what you really want to do is up level all of that and that's what tetragon is really good at doing taking these low-level data. So you can monitor all your files monitor all your execution, create a nice execution trace for anywhere in your cluster, along with timestamps into a data stream right.

B

So now you can analyze all of that data right. You can ask questions like what executed three days ago on this system and what programs did it run? What files did it open? Who did it connect to this question?

B

um You might even ask you know what are the DNS entries for those IP addresses that I saw these connects for and what's really interesting for today. What BPF programs did it launch right?

B

um Because that's what we want to answer so when we talk about BPF, the next thing we want to talk about is what is a BPF program right, so typically I think of an application program, you're thinking of a set of a like an executable, something that you launched, but BPF is slightly different for one that runs in the kernel. That's also interesting, but what is kind of interesting from the question of what's running is that BPF is not just a set of instructions.

B

There's an entire run time around it, so you can think of it as a set of instructions. Plus all this other stuff and the other stuff here include core. What core does is when you load your BPF program, it's actually a runtime rewriting kind of framework, and so it takes the program it loads it in you're, going to replace a bunch of the instructions with kernel specifics.

B

So if you think about I want to read a structure, a field in a structure, maybe the PID of a task or something- and when you do that that PID location, that structure, you're going to say, read the task struct and then read the offset into that task. Direct is not the same across all the kernels, so what it will do when you load your program, but the core infrastructure does is rewrite those instructions so that they actually read what you're interested in.

B

So for one, that's interesting, because now your set of instructions that you downloaded onto your program, it's not the same instructions that you actually run. Okay, um just one I one kind of point on why your BPF program is sort of different than just a set of instructions. In addition, you have a bunch of maps, so the maps are the piece between BPF um between the kernel and user space that communicate between the different programs. So your kernel piece might write to that BPF map, but also multiple user space applications might write to that map.

B

Are multiple kernel pieces might write to that Spectrum. So, if you think about it, it's almost like a um like a message: bus between different programs and applications. It can be memory mapped, for example, so that influences what map that program is attached to will influence kind of what the system is doing.

B

um And then you know there's what are you connected to? What type of program you are and so on um and all that together, in my mind, is actually what you want to monitor. You want to monitor that bundle of BPF stuff, because if you imagine a BPF program, that's connected to the wrong map is going to be buggy or Worse malicious, a BPF program, that's attached to the wrong function, isn't going to work either in the way you expect, so you really want to get that full full profile.

B

um This is a diagram that Brendan Gregg made and then we modified it slightly Daniel borkman, um one of my colleagues also modified it and what it shows. Is, That, Flow graph of what loading a BPF program involves, and so on the right side. You see the application side. What is the BPF application or sorry yeah left side? You see the BPF application and what we have over there is like the instructions like we talked about.

B

You have a compiler because at some point you need to create probably C code into bytecode um and you create your maps and all this kind of stuff. And then what we see between the left and the right side is the boundary between the um between the kernel and the user space slide, and you see a bunch of different objects. You can modify the bottom there. You have perforing BPF ring buffer shared Maps.

B

Perforating is just a high performance way to get stuff out of the kernel between user space and kernel, shared BPF, Maps or any of the other map types. We have hash. Maps array, Maps Stacks, so on people are always adding new things there and then the top is where you actually load your programs, which is actually done through a system call and um and loaded that way with an opcode.

B

So when we think about monitoring this, the question is like: how do we want to monitor this? Well, we don't want to hook the syscalls directly, usually there's a good reason for that, one of them being that we don't want to hook every possible syscall that interfaces with BPF, and so what we do is we hook the BPF verifier. The verifier is the piece that, when you load your program, it's going to make sure your program is safe to run in the kernel.

B

It's going to make sure it's not writing to random data, trying not to make sure it's not reading random data. It'll also do other things for safety uh check for divide by zero. Those kind of things that you would think of what should normally cause seg faults in normal programs. So there's one one reason to put it. There is just because it's the central place where all BPF programs come through the other advantage of having it. There is it's after the user to Kernel space copy.

B

Okay, so one thing about BPF: if you hook a syscall, what can happen is if you try to read user memory, which is just a pointer into user memory, it's entirely owned by the user space process. At that point right so from a security standpoint, we don't want to try to read user memory and race with a user right, because a user owns the memory they can. They can change the data right.

B

So if we're trying to do a security property- and we say I- want to read your instructions or read the name of your BPF program and I'm going to say, do some some analysis of that program from the BPS side or even, if I'm, going to copy it out to user space for sort of post analysis. I need to make sure that I can't do that copy and then have the user just change the data right there's also, a more fundamental reason um would be that user space can actually just fault right.

B

You try to read user memory, it's not in um it's, not in Cache, it's going to fault and then because BPF on most instances does not want to sleep, will not be able to pull that fault into the memory, and so what you'll get is just an error. You'll get no data, and now you have a big glaring Gap in your um in your security analysis tools, presumably running behind this.

B

um So that's why we hook BPF verify the summary: is we see every BPF program? That's loaded! We've put a BPF program right on that call anytime. You load something we'll get a call back and then I'll talk in the next couple, slides what we do with that callback, and then we also hook these other entries. You have to be a little bit careful for the syscall reason, um they're actually done on the other side of the copy inside.

B

That was this call, so they're, not technically ciscalled, but in this box we just show them on the edge there um and those are for all your maps. And the important thing is: um if you look at that, dotted line everything. That's going from the user space to Kernel space going across that dotted line can't get to the core networking part in this example. But we have other slides that show kind of K probes and other things can't get to the far right without going from one of our red boxes from left to right.

B

Oh okay, so the the text in red is where we're going to put hooks um BPF hooks to capture the BPF loads, and so the um sorry, if I, wasn't clear. So when you look at like this flowchart going from left to right, we want to make sure that we go through a red box before we get to the far right, where the programs actually run to ensure that our analysis, tools that are running in BPF get an event that something happened from the from the application to the kernel side.

B

Is that clear, great perfect thanks? Sorry about that? If I didn't say that up front, so then, once we get this flowchart instrumented with BPF, we now have a way to get these events. So then the question is like what are we actually looking for and what do we care about so the first case in our kind of cartoon picture here is the good case where you have some application: I call it Alice here um that application probably has a hash associated with it.

B

That's like a shot 256 for the executable I called it Foo here, but you know, usually it's 256. A shot 256 or 512 or whatever you set up your system for and then it has a program and some maps. So alice.o is the BPF executable here and then I call it map a and map. B are those two maps that let you push data between kernel and user space.

B

um What that application is then going to do is cause that call that CIS BPF call. That's the system call to load a BPF program. If you go back a slide to this, what that's going to do is cause that BPF loader Block in the Middle top middle there to call the BPF verifier with that alice.o, which is then going to trigger our event system so that we get a notification that a program has actually been loaded, okay and then on the bottom.

B

Here, as I put what our system spits out at that event, we see that event. We say: there's a BPF load happened.

B

um This is annotated with the kubernetes namespace, so we put the kubernetes namespace there BPF and S the Pod Alice in this case, along with the application that loaded it Espin Alice, we can put the shot 256 there. If you have the system set up for it, we gave a talk at kubecon about that. So I won't go into details there.

B

We give you the program type, which will tell you what it what kind of program it was if you're familiar with BPF, there's K probes and there's networking Hooks and there's security, Hooks and all this kind of stuff. So you kind of want to know that as well and then we'll give you some details about the function about the instruction set that was loaded. They have a name Alice function here, extraction count and because with BPF program now has access to the entire BPF op.

B

As it's being loaded, we can put arbitrary other data there as well for pretty printing pick some things that we think are as interesting, but we actually have access to the entire BPF instruction. So you can do things like copy that program out so that you have a copy of every program, ppf program that was loaded.

B

They can be kind of large 4K instructions, but you know one page: BPF loading BPF programs is usually not something that you're doing very frequently so doing a copy of 4K bytes is usually negligible, but it depends on what your use case actually is. So this is the good case. Program gets loaded. Events happen, nice, nice, pretty, printing of the auditing flow.

B

So let's just explode our example slightly and to check my time explode our example slightly and we'll add Eve and Bob here and of course they have their Shaws and I think we called it um bad and bar there.

B

So the next question is: what happens if Eve tries to load something we don't expect Eve to load something right well just like before, since the since the tooling doesn't care right, it's going to give you the event for the two, but what's actually a little bit extra interesting about this is I.

B

Think you would make the argument that if Eve is running in its own pod, you probably should have never gave it cut BPF if it wasn't meant to load BPF programs right like basic uh basic capabilities here for pods, but what's extra interesting about this is even if this eve happens to be running in Alice, which you gave cap vpf, because you needed Alice to load a BPF program. You still get the audit so if, for some reason something inside your pod is loading BPF programs.

B

Besides what you expect to you'll get an audit record for it, and the next interesting thing is with tetragon we have enforcement and which allows you to say match on binaries or Shaw 256s. If you want it's easier to look at it's easier to look at um uh program names than Shaw names, what you can say is if I see this call and it's not Alice, stop it from happening right. So, even if you're in the same pod, so you need to have cat BPF. But you know exactly what program should be loading BPF programs?

B

You can encode that into your policy. That's a crd and tetragon world and then what happens is if anything else inside that pod tries to load the program under BPF program, we'll block it out right, an extra layer of kind of protection from things trying to load, BPF programs for you um and of course, like I mentioned, you could also use Shaw's, but that's a different talk requires a little bit of extra setup.

B

um So the next question would be what happens if you have a file system problem where you have, since Maps inside of BPF are usually filed, descriptors, but they're always file descriptors. Sometimes those file descriptors are penned into a file system, and so you might say well: Eve tries to access a file from Alice and this could be like a policy file. It might have sensitive data in it and so on.

B

um Well, your first layer of his fans should be well. Don't let Eve actually access the file. You know maintain your amounts, but if you can't do that for some other reason, tetragon can also monitor file access, so you'll be able to see which BPF programs access which files those files are to Maps. So you can see, Eve is accessing map B there and um again you can enforce or you can just create an audit Trail for it.

B

B

This is so that um if this is the other discussion, we're talking about Bob loading, BPF and I'll just skip this one I think for now. So um in summary, what we've covered then is Alice is the good case. Alice loads, a program we get a nice audit log for it, Eve is the malicious program or at least erroneous program tries to load something gets blocked, even when it's in the same pod, because we built this policy um and then we have the file monitoring.

B

So there is one case that I want to call out that the um we're working on inside the BPF foundation and there's kind of a working group around this if you're interested, is what you can say.

B

Well, what if Alice or Bob in this case is allowed to load BPF programs, but they load a different BPF program than they are supposed to meaning you launch Bob, you checked, you did some Integrity check to make sure it has the right Shaw somewhere along its lifetime, it decided to load some random instructions that are not the instructions you expect right, which it's allowed to do, but these are perhaps instructions you didn't want it to load like a malicious program, because the maybe Bob has been compromised somehow um so there's a couple things you can do.

B

One thing we've suggested is: if you just keep a log of all these app these programs, since they're loaded copy the instructions out at least you'll know what programs are loaded and you can kind of post analysis and say this application is loading this BPF program? It's not the one I expect it's a new one, I've seen I've never seen this one before. Please fire an alert off to my alerting tools.

B

The there's a working group also going on about how to get the right, um how to put a Shaw on that BPF program itself, and this is actually more difficult than it may sound on the surface, because if we go back to when I talked about the core that that application is not static, it's being Rewritten by the runtime. So if you were to just take a shot of the program up front, it wouldn't match what the Shah was after the program's been Rewritten to load on your kernel.

B

So um what we're doing is trying to get the kernel to do a lot of these rewrites and have sort of a consistent Shaw before and after the modifications um find me afterwards. If you want to talk about it, it's really quite interesting, but it's it's a kind of a work in progress. I, don't think anyone has it fully deployed. Yet, although some of the newer kernels can support most of the base functionality for that and I. Think in summary, like you said, Alice is good to load.

B

We get an audit, we can block Eve, we can watch the files and then we got a plan um at least to audit the the kind of erroneous program, loading, bad instructions um and then in the future, we'll have a full solution for signing the VPN programs.

A

Yeah, so in the last 10 minutes, five minutes, I will just show you like a quick demo like how could we actually do it with um tetragon, so how we will do it? We will introduce a test environment, apply a security policy which would actually observe BPF program, loads and the map creations. And then we will have like a simple use case just from raiko test pod, it's called BPF Droid and then the last one will be.

A

um We will actually see like what kind of BPF programs clu loads and then what kind of BPF map serium craze during its boot up. So how the test environment looks like it's going to be a one node GK cluster. We are going to have tetragon deployed on it as a demon set, and then we will apply a security policy which would generate events during a BPF program loads and then BPF map creation.

A

So, for example, if you would have like a multiple node cluster, then you would Deploy tetragon on each node as well, and then you would use a grpc collector to observe actually all these events.

A

So this is how actually the policy would look like. uh We are going to observe like three main kernel functions, BPF check. So this is when the verifier actually checks the program before loading it. We have security, perfect event unlock. So this is actually creating a map um to transfer events between user space and the kernel, and then the last one is security. Bpf metallic. This is basically um when we create a BPF map, so we have a simpler pod and then what it's doing?

A

It's actually loading uh BPF programs, yet a BPF, prog and then via bpf2, and then it's going to sleep for 30 seconds and then it's going to create a map, a tetragram BPF map with hash type. And then it's going to sleep for 30 seconds as well, and then this is actually how the events look like. So we extract it as Json events, and this is just a CLI like 3D, printing it and then basically, in the first row, we can see that the BPF tool probe load process started.

A

We can see the kubernetes namespace and then the Pod. We can see Washington namespace and the Seattle BPF Android mode and then on the third row. We can actually see like that. Our program was loaded. We can see the function, name, the amazing function and the instructions and then a couple of row later. We can actually see like when the map was created, so we can see like it's tetragram BPF. We can see the key and values and then we can see also the um hashmap type.

A

So for cerium it's a bit more complicated. So I will just like highlight certain events and steps that are that are executed during the boot up.

A

So it's going to grow probe a bunch of features and maps that are available in the curler version and then it's going to check and remove some IP tables and then it's going to execute a script, um basically figuring out like which programs to load and then what network devices to load them on and then basically it's going to compile those programs and then basically it's going to load those programs for each pod on the Node.

A

So basically, this is the screenshot um that was captured by during the demo and then we will see it alive. So these are all the programs um that are going to be loaded for each, but on the Node. So we can actually see here, for example, um an example like a program named sand drop node. So this is basically Bambi send notification. Whenever a packet was dropped, we can actually see how many instruction did it count. We can also see the Terra call handling programs.

A

um We can also see policy handle, so this is related to how psyllium handles actually um Network policies and so on. So I will just switch the terminal and show it um live if it works.

A

All right so I'm just connected to the GK cluster and I, should have tetragon running on it.

A

All right here we go, so let me just start to observe the events related to the um to the BPF trade, the test spot first and then after I will show the celium use case as well.

A

All right so I will just apply the pod.

A

And the money one seeds get it gets created. We should see some events as well all right. So basically we will see like the bpf2 progload. You can see the program uh program type here, um like BPI prototype crypto. We can see the functions and the instructions and then we can see actually the sleep and then we can see also the kubernetes information here, like Washington namespace and the Seattle BPF Droid dot source code, and then it's going to stay for 30 seconds and then we are going to see actually the um the map creation.

A

So it's BPF mapa log. Then we can actually see the BPF map. uh The hash.

B

A

It's a hash map and then we can see the type here and then we can actually see the name, the map name and then basically a key and values and the max entries.

A

So I will actually just stop this and then I will start to observe the events from psyllium I actually have psyllium set up on on a different namespace. It's the celium namespace.

A

All right, so it's actually running on the celium namespace, so I will just restart the agent and then we will see all the BPF program loads and map Creations. So it's going to load or going to probe a bunch of features like what features are available on that kernel version and what BPF map types are available on that corner version. So um we will see a bunch of Proverbs in the beginning.

A

Okay, so we will see actually the termination we will see the put up.

A

So these are all the probes that cilium is actually like running to figure out like what features are available and what what map types are available. We will see some IP table rules like figuring out, like what um delete delete, so I will just like um highlight the ones which are related to program loads.

A

So there is always like a recurring pattern here and then it always starts with like um filter, replace and then basically it ends when the command exits. So, for example, these are what the BPF programs that are loaded, for example, for each pod, and then we can see, for example, the center of notification here and then, for example, the terracott related ones, and then we can also see, for example, handle policy which is like how um psyllium handle policies and then basically it just does it for each board on that specific node.

A

So this is, for example, like um audiover the example like how could we create, for example, audit list for each programs like um from each pod from each binary and then basically keep track of like what has been loaded, and should we expect this um or not?

A

So let me just switch back to these slides.

A

A

All right so as I wrap up like ebpf, is on the rise, and then we see more and more applications for networking, observability and security.

A

um It's basically cross-platform and then what is important to keep track and that would it like um who is watching eppf like um reach programs and Maps were created by who and when and then. Basically, we saw an implementation on how tetragon could do it. So a bit more on like how to contribute.

B

Sorry there we go um how to contribute yeah so join the join. The slack Channel come to the GitHub page and um what else did I put up? Sorry, um oh we're always looking for more use cases. So, if you do, if you go to the tetracon repo there's the crd's examples, um you know they're sort of incomplete list of things that we care about. But you know mostly it's things that I thought were interesting, but either contribute or if you just file an issue and say like I. Have this use case.

B

um The nice thing about getting them in there is that we generally run those before we do a release. So if you're, using tetragon for something or you want to use tetragon for something put your use case in there and that's a really good way to make sure that we know if we break it somehow even better would be to write a CI test case, but hey we do run those before we release. So that's kind of your first gate, um yeah and any feedback. uh Let us know to find a bug.

B

Let us know, of course, we always always fixing things and adding new things. So if you have a new use case, let us know- um and you know, documentation and all that good stuff would be- would be great. So thanks a lot.

A

Okay, thank you.

B

Any questions right, we.

A

Have a few minutes for questions.

B

If anybody's got anything, if not yeah,.

B

Okay, I'll, just repeat the question: the question is: does tetragon use bpfl at SIM for anything um to preemptively block things? No, we do not use BPF LSM. We do preemptively block things, though we have a synchronous way to use um to kill processes. So, instead of blocking the call, we just kill the process if it's violating the policy.

B

No, the action would not have gone through because we do the Sig the the send happens in in line in the kernel. So we kill the process from the BPF program, which will then cause all of the cores on the application to see it as a terminating signal that it cannot block because it's a Sig kill, so your application cannot catch and block a Sig. Kill from the kernel is from a BPF probe yeah.

B

So there's a BPF helper to do it yeah yeah, and so that does it uh there. There is uh some slight caution. You can't do that from anywhere in the kernel, but when we deploy this, we ensure that we do it from the right places in the kernel. So it requires some care when using, but um it is doable.

B

Yeah anything else.

B

Maybe I just that's where that um when you looked at the BPF when we stopped blocked the BPF program, that's that's how we did that.

B

Anything else cool all right. Well, thanks for coming yeah.

A

Thank you for coming.