Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Binary Authorization in Kubernetes - Aysylu Greenberg, Google & Liron Levin, Palo Alto Networks

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Binary Authorization in Kubernetes - Aysylu Greenberg, Google & Liron Levin, Palo Alto Networks

Kritis is an open-source solution for securing your software supply chain for Kubernetes applications. Kritis enforces deploy-time security policies that ensures only trusted container images are deployed on kubernetes to your cluster. With Kritis, you can require images to be signed by trusted authorities during the development process and then enforce signature validation when deploying. Kritis enables tighter control over your container environment by ensuring only verified images are integrated into production. Talk outline: - Introduction to the concept of binary authorization - Live demo of using Kritis and Grafeas for deploying images with confidence in Kubernetes - Grafeas and Kritis roadmap At the end, attendees will gain solid understanding on the process of binary authorization and how to incorporate it in their build and deployment pipelines

https://sched.co/UaZl

A

Good morning, everyone welcome to coop calm, San Diego 2019 day how's everyone doing today this morning, yeah awesome awesome got some coffee, I'm very caffeinated. So so, let's get started today, we'll talk about binary authorization in kubernetes. How many of you familiar with the term binary authorization raise of hands?

A

Okay, like half the room, good good good for the rest of you, I'm excited for you to learn about what this means, and hopefully this will help you in your work clothes. So, let's get started a nice little Greenberg I'm a senior software engineer at Google throughout my career, worse than a bunch of infrastructure projects such as search infrastructure, developer tools, infrastructure, Google, Drive infrastructure and these days, I work on Google cloud platform, infrastructure and I'm.

A

The engineer of two open source projects, graphics and credits which we'll talk about today- and you can find me on Twitter and I'm today, I'm presenting with Lee Ron Levin, who is the chief architect at Palo, Alto Networks, and he is one of the biggest most valuable contributors to both graphics and kritis projects. His contributions were instrumental in launching zero point. One point Oh, Dreyfus and Critias and he's been just a huge help throughout the whole time.

A

So today the talk will be in four parts. First, we'll talk about why we need binary authorization kind of tomato bit. Why we even talking about us today, then we'll talk about how to improve the security posture of your kubernetes cluster, then we'll talk about exciting new, open source projects, graph is and credits and how they fit into the need for binary authorization in kubernetes and, finally, we'll have some fun and see a cool demo that they're unprepared. So let's get started so how many of you familiar with the term software supply chain?

A

Okay, a little more than half of the room, great so for those of you who are not familiar. Software supply chain basically describes the whole flow of the code from the time the code is submitted by the developer to the code base, and then it's picked up to be built and tested by the continuous integration pipeline and and deployed to kubernetes by the continuous rape. I plan okay. So now that's the kind of the you know they in syria. How it's all lined up beautiful in tonight step? So, let's talk about the reality.

B

We have in our state environment I have if you installed at kubernetes, is told- and I from simple application now when I look at it like I started to ask myself several questions so just by looking at the class they're like do I know really like what kind of images are the now to my my testing.

B

Might aircraft and do can I tell if all those images will test the required QA testing, Oh, DevOps, testing, okay and is one of those images are impacted by a severe security vulnerability and, for example, the execution with cost millions of dollars now, just by looking without doing anything like I. Don't have this information about my cluster? So what can we do? So? This is the sort of supply chain we saw, but maybe we're missing an accession component, okay, that will authorize and verify stuff before they get inside a production cluster, okay.

B

So this is what we want to do. So what are the use cases for binary authorization? First, we want to ensure that images inside our tests last about production cluster will be signed by trusted authorities. Totus can be if you a DevOps and it can be any security tool, but you want to ensure that somebody approve an image before it goes inside production. Second, you want at your image Apple pests from restrictive security carrier, for example, no critical civility and patron ability; okay, you want, will ensure that.

B

And finally, you want to continuously monitor your inventory in data inside of production cluster and verify those kind of security issue. Doesn't react, oh okay, all the time. So luckily there are two project together can implement and fulfill those and solve those problems, so first Grazia's, which is an artifact metadata API in technical terms. This is like a way for you to put information about stuff that are running inside of cluster and second kritis, which is a deploy time. Policy enforcement, a solution for kubernetes, so choose nothing.

B

Vulnerable or non-compliance, will get inside your cluster so to understand how it work, let's go over together or at a very simple example of creating a pod in kubernetes cluster with critics and gracias. So, as you all know, first wave the kubernetes api. Ok, so when you do cube, cuttle create both a cube, cuttle a called equivalent. This api, which are edited HTTP requests, then we have created, which is an admission control component for policy enforcement. So creepies is an admission webbook for validating request.

B

Ok, this is a standard, a standard, its component inside kubernetes, so kubernetes api call treaties to validate the pod creation request. This goes to a validation we cook, which next call to the image security validator component. Ok, so this component is in Schappert stupid. They thought that you're going to create. Ok, it's compliance to the policy. Now you ask: where is the policy so in crete is the way we implemented? It is to put the policy as a CD a custom list on technician.

B

So kritis policy is the theory, meaning that, like we use field, is an extension to kubernetes. Well, it enables you to put like yo kind of like your policy or anything inside kubernetes and quick when it is to fetch them so to understand exactly what is the security policy? Let's quickly go over an example create policy? Ok, so a clear! This policy has a kind in this example. This is an image security policy, because we've got we want to verify the security of an image. Next, it has a at the station authority names.

B

Those are the authorities that can sign other images and that can look like a poof images that are going to be deployed inside your cluster. Ok, we have an image allow list now, usually we don't want to run security checks on all the images in the cluster, for example, for infrastructure imager like sto oo, stuff or images for communities, so you don't want to verify them for vulnerabilities, so we also enable you to whitelist them next. For the important part, we have a requirement for a full package. Vulnerabilities.

B

Ok, so see this gives you two types of properties that you can specify and to to to this is specified, fell namespace, okay, pail kubernetes namespace, so it gives you the ability to say what is the maximum civility of maximum stability, of a vulnerability that can be inside one of your images. Okay, in this example, the maximum civility is high, meaning anything I above high will be rejected.

B

For example, critical security, vulnerabilities, the second property- is the maximum fix unavailable stability, so not all vendor patch or vulnerabilities, maybe because the vulnerability is low severity, maybe because it's not applicable or not exploitable. So what would you do when, like you, have a vulnerability, but you don't have a way to patch it, because you an upgrade of the package, will not work so for this we provide you this property with just define what to do for those cases where there is no solution for the vulnerability. Okay, no way to update it.

B

Finally, because it's not best practice, you can purge all runner abilities in the world. Sometimes you don't have enough time. Sometimes there is no solution for the vulnerability, so we also enable you to specify a whitelist of both offense series of vulnerabilities, which you can say, I will fix them later or I will not fix them at all. Okay, so now that we have a good understanding, look like a so we get a policy for API, and now the policy is inside the image security validator.

B

Now, in kubernetes there is no notation of vulnerabilities Oh data about images. There is also only the specification of the poll. So what do we get this data? So for this we have gracias. Now, Garcia is an amazing project. It's a community driven a project to like have a uniform way to have metadata about stars that are running inside your cluster. It can be images, vulnerabilities deployments and much more so, as I said, it's a uniform way to audit what goes inside the software supply chain.

B

Everything cloud native goes inside gracias and it has a common protocol and an API to do all that. So, let's shortly understand how graphy has metadata store works, so in gracias. The first thing we have is it's called nodes, ok and think about it as specification that scribes a piece of metadata- okay, for example, vulnerability- is a metadata, so it has a kind okay. It has a detail, for example, which package is impacted by the vulnerability. It has a mean affected version, so from which version this vulnerability is applicable. Okay.

B

So this is a note and there can be many other types of not. It can be an attestation of told you note, meaning. This is an entity that can sign stuff in my cluster, okay and it can be deployment and can be Billy Stewie and can be many many other more properties that are continuously expanding. So those are things that security tools studied kubernetes and other tools can use to understand information about stuff that are running inside your cluster, okay, very cool.

B

So the second thing we stolen Garfias is occurrences and occurrence is the actual instantiation of a note, for example, vulnerability. So vulnerability has no mini bytes that it has meaning, but it doesn't have meaning by itself it's not applicable to anything. So a note is the applicability of something inside your cluster, for example, ability. So it has a kind okay and it has a resource and the resource. In our case, our vulnerability is the image hash.

B

Okay, so you can say this is vulnerability, impact this specific image, and you can also specify how this vulnerability impacts the image, for example. What is the affected version? So in my image, I have a specific package installed, which a specific version and I can say this inside Garfias and next I can also say what is the fixed location. So what will happen if I up get like what is the minimum version?

B

I need to upgrade my package to be non vulnerable okay, so this is like how we specify a vulnerability in inside graphene's and it could be. There are many other types of occurrences it can be at the package. It can be remediation and build this to II and many many other interesting artifacts. Okay. So now that we have good understanding what is graffia and what is this all so the image to keep our data inside treaties calls the API to fetch the metadata okay and in the open source implementation.

B

We and again is existing open source and you can use it so in the open source of implementation of graphics. You can. We have three types of databases that you can use. You can use in memory database. You can use a embedded database using vault DB or you can use Postgres okay. So all of this is already implemented, so we take the data from garages. Okay now take. The handling question is who pushes security data to Gropius, and the answer is security tools, for example, twist lock.

B

Oh, please, no public cloud, so we scan your images or scan your hosts or scan other other cloud native components and push all these data to graphics, and there are many many other tools like container security analysis and many many other efficient tools that can push data to graph. Yes, okay, so now that we have a good understanding of how graphics works, let's see a demo. Okay, are you ready for the demo? Oh good?

B

Ok, so the first thing I'm going to do is to install graph. Yes, now the dem I'm going to show you a it's already written. It works perfectly. It's inside the cletus github repository. You can use it today to install a for instance of graphics and critics using help chart, and you dig in totally on in your JK class tell but very soon in any component after that, you're using okay. So everything here works, which is good. So first thing we're going to do is to setup grass, okay, so fill it in to the walk.

B

B

B

Okay, so this, what do you do now? It's just run a very simple script that install graffia is using the hand shot in my jakey cluster okay, so we can see that graffia is running okay, so we get the port and it's in the default namespace book, and you can obviously modify this just for the demo. So we installed a grass fields now. I have a database that will store information about stuff that are running inside my cluster.

B

The next thing is to set up kritis again another hand, job that will install quality equities and will automatically connect it with graphics, very cool. So again, it need to create a certificate to create trust between critics and guardias, which is fine, I'll take a few seconds in turn public. My problem and okay, that's good! Okay. So it's funny excellent now I have rapiest I've quit is what is missing, so I need to set up a security at the station to enable kritis to sign over stuff, so that, like it knows, who did what?

B

So, let's set up a let's sorry.

B

Okay, so let's create this, the station authority, so I need to create annotation authority to enable quit is to sign of images and information. It's going to be deployed, and, finally, okay, so I'm going to deploy the policy. Now this is the policy I'm going to deploy. This is the same policy. I showed you a few minutes ago. It has the quickest authority, it has the maximum stability. The maximum stability po4 in this policy is high, meaning something with critical severity will get rejected.

B

It will allow any vulnerability that does not have a patch, which is it's not really about security, but this is what we're going to show you today and that's about all so, let's create which cube cartel. Let's apply the policy, okay cool, so we play the policy it works. And, finally, okay, now we're going to scan some images I'm going to scan two images, Jenkins and Ingenix. Now Jenkins is good and they have a huge security pipeline, but I'm going to scan a very old junk in the instance. So let's scan the images.

B

So what you? Okay, so again, the images using Twitter lie. Twisty lies twist clock tool for a scanning, a braunability's inside images. So, as you can see here, this is the result and they don't look good, because I have like very largest room, but I have six critical. A vulnerability is inside Jenkins. Okay and I have two medium vulnerabilities inside Ingenix. Now, according to the and this tool, also publish everything to Gladius the nodes and the occurrences now according and the data that I pushed. If now I'm going to apply a pod, the nginx pod.

B

Okay, will you think this will work based on the lecture? Yes or no? No, you win I mean. Will it get rejected? All the food I prove okay, so Angelique's approved good okay, but doesn't mean anything because even without us. So, but let's see what what's going to happen when we do it with Jenkins, a port object: okay, okay! No! Yes, it's rejected because a a Mission Control quit is validation. Well book denied the request because it found relation.

B

If I will look at the logo, we see all the evaluation that happens and I can also look at graph. Yet a whole graph is API to verify or devaluation and see what what's really going on and what can I do to fix all that? That's good and that's in tell them: oh okay, so it works. You can use it today and that's.

B

C

A

Demo work: it's always the scary thing with live demos, but so what's next for graph isn't credit. So let's talk about the road map so for graph is first of all, we would like to add new metadata currents that help you express for your CI CD pipeline for yourself for supply chain, all the necessary information that you could store. So some of the new metadata currents that were proposed by the community, our licenses test information which test pass failed for any given image or the fact static analysis for any kind of static analysis.

A

You would like to run on your code and record that information, and also in total link at the station periodically we'll have communica where we discussed this with the community. We discussed the representation so on so, if you're interested in contributing more metadata cards or commenting on this please tune in in the community. Calls now also we'd like to move our server zero point. One point I believe it's two now the micro version would like to move it to 1.0 version and it's a little thing about the 1.0 version.

A

So basically, what does it mean for it to be one porno, its production ready, it's proven, but in production that it's working internally, we'll have our own and Google, and so the open-source version would like to get it there as well, and this means using up the view on produce and also ensuring that it's just running and stable, and you can trust it now.

A

We would like to move towards larger community contributions, community participation in this, so this means that we'd like to have community members step up with paint a nurse for client libraries for each of the languages. Currently we have libraries for Go, Java, Python Ruby, and if you are using those languages, most likely your the most knowledge about about all the different pitfalls or the different edge cases which it's not quite working as expected.

A

So if you'd like to step up to maintain them and make them work the best possible way, then we'll would welcome that and also we would like to have the reference server. Implementation moved to 1.0, with the community help as I mentioned internally. We use our own implementation of graphics and I use. Sometimes the the standalone version that leerin presented in the demo today, but those of you who be using it all the time would be a lot more knowledgeable and can contribute a lot more different ideas on how to make this robust and stable.

A

A

What about critters so for critters would like it to become production-ready highly available critters by that I mean if graffiti is unavailable at any point, making sure that critters can still either allow or reject your pods and have a very well-defined fallback policy that you, as the user of critters, can define yourself and also providing an ability between binary authorization on Google cloud platform and the open source kritis so making sure, if you're a user of the Google clouds project, and then you start using open source or vice versa, that you get a seamless experience across both and binary authorization team is working on this.

A

There is a zero point. 3.0 version coming up where this work is being done to ensure interoperability. And finally, we would like to have more expressive policies so that you could use any of the metadata kinds stored in Gracias and then be able to write policies based on those and have the deploy policies, deploy verification based on the metadata. You actually care about whether it's you know ensuring that build was assigned by the appropriate entity, whether you want to make sure that at least 99% of the tests have passed or the most important tests.

A

You know, maybe all the unit tests, but some of the flaky functional tests are ok to skip and so opportunities are limitless and all the different combinations are infinite here cool. So if you're interested in learning more there is the tutorial on deploying standalone critters with Stalin graph is that you saw in this demo today, and so you can start running it today. There's a the gracias project is developed open-source first, so you can find our github greatest project. It's also developed open source first, so you can also start contributing to that.

A

There is a three mailing Lillis that we have, which are gracias users for anyone who's using a gracias there.

A

If you're interested in contributing to Brad Reyes as a developer as a contributor, maintainer and then crudest users, so please join those mailing list and stay and update up-to-date on what's happening in that community, and we also have a Twitter account for any questions or any kind of big updates that we have so to conclude, I would like to ask a question how many of you are interested in either trying out or can trying out critics in graphics, first, okay, how many of you interested in contributing of stepping up?

A

As my turn, okay, nice, nice, very good? Well, thank you! So much my clicker stopped working at the Brewster. Oh.

D

A

Maybe No, thank you so much.

A

And let me check how much time we have. We have about 10 minutes to take questions. So, yes, lots of hands leti. Let's start from right side to the left and we're just kind of scanning. Okay I doubt go ahead.

A

The question is: do we plan to donate the project to ciancia? So that's a great question and we've kind of been asked that for the past year since I've stepped up as the maintainer of the project, it is an interesting idea right now the project is just very in very early stages, who are still trying to figure out what is the most necessary thing for the community, so we are considering it, but it's probably not going to become urgent anytime soon, just to figure out the fit. Okay.

A

C

E

I bet, you said so on Opa couple of sessions and I see a little bit of overlap between what krafayis and is doing very piecey. Please go and use them both together or it's actually two different concerns.

A

Expert question so a great question so, for especially the last part dimension about them, their roadmap for more expressive, policies that something that has come up and we are evaluating actively what their OPA is. I get said it has been brought up by the several members of the community. So it's definitely in the back of our minds. We're thinking about it and if any one of you are experts in Opa would welcome your contributions and your ideas for this. Thank you. Okay, more questions and most start scanning.

A

Okay, you in the back in the black shirt and.

F

If you had the ability to add exceptions based on like a sub package within within an image.

A

So, just to repeat the beginning of the questions for those of you who couldn't hear it so we have some allow list, the exceptions and image security policy. Could we add more of different types of allow list again open to that and that's definitely something that I could see being useful to the community to the user.

A

So if you have specific use cases for which you would you know, definitely going to be using it, please chime in open github issues with the description of what you're using and then we can discuss it with the community and add that definitely open to modifying this and adding new types of policies in the back place.

C

G

So I think group is is great, but one of the problems is like this is all data right like this is kind of what's going on in your system, whether like, like your bills and deploys and stuff like that, but is there any kind of talk of it's like a dashboard, some kind of visualization and I've actually like pulling together all this information, so you can have conversations about it other than going to you know any of the commercial vendors and having to use like their products that kind of wrap around orpheus mm-hmm.

A

Regression, so maybe specifically, the focus of the graph is, is to provide the API and the spec of the metadata kind and then have all the different vendors in the CI CD pipeline, integrate with it, and so the scope is specifically around providing good, solid api that is easy to integrate with it's also agnostic of the storage backends.

A

So we don't dictate what storage backends you should use, and then anyone who wants to be able to go on top of that they are welcome to the thing is that different gracias, including users, will have different needs for the kinds of queries they would like to run on top of this, so graphics, API is just you know, very a straightforward, crud API for retrieving notes and a current and ask you some questions and then, if you'd like to write higher-level queries on top of that, that's something that is on a case-by-case basis.

A

But again, if the community finds common needs across the boards that should be implemented, there's definitely something you should be discussing. Thank you.

F

Sorry, I noticed the tool is effectively validating before deployment of the pod at the kubernetes at potentially the customer site. However, it seems to be driven by a text policy. Is the text policy in any way itself protected against unauthorized modification that might allow a pod that should not run to run. Thank.

A

You for the question: do you want to take this? There are.

B

It's like you can prevent this with this person's photo ization like user. This can touch that type of it. But again it's not like a complex. It's a concern for kubernetes that an otherwise authorities will modify policy. My.

F

Company that produces a product that ships is a large number of micro services and I'm looking to validate the provenance of our micro services, all at the customer site and many develop team can produce micro services, but their micro services should not appear at the customer site. Only the officially tested, validate or whatever, and it seems like I with proper knowledge, could fake a micro service that goes to the customer site. That has a policy that lets it run. Even though it's really not something officially built.

B

Kubernetes position because the objects here, the namespace. So if you have like misconfigured your kubernetes configuration to allow users to modify like next visit, they shouldn't have well.

F

No I was thinking if the policies themselves were signed and you know checksum, so that could be no. You said no, this. This policy doesn't have the right key. Then we would ignore it. We would fail outright. You've got a bogus policy, don't use it so.

A

There's other ways to look at it. So one way is at the crudest policy right there. Customer resource definition, which is you know, exist as it is it's an extension of the Correa's API. Another thing you could do is store other with existing metadata currents in gracias or add new metadata currents. Were you basically signing those things and requiring all the different CIC deep pipeline steps to sign in and verify it so before even gets to deployment step, it would be verified along the way. Thank you questions. Let's move to this side of the room.

A

Yes, you in the back in.

C

D

C

D

On your roadmap for adding support for other cloud providers or on-prem kubernetes for criticism, great.

A

Question, thank you. So I will welcome community contributions. You know I personally, just working on TCP infrastructure work a lot with jke but again very open to community contributions, whether it is to run a mini cube or any other communities. Implementations.

C

H

But what's gonna happen box meditation? Is it good? Is it gonna install this automatically in my cluster because I know there is like a checkbox one rating that you gave could.

A

You repeat the question: please: it went in and out.

H

Sorry so, while creating the GK cluster in the console, there's like a checkbox for binary validation, just wondering what's gonna happen under the hood when I just check this box right so.

A

I believe you're talking about the binarization on the GCP, which is the the internal like. The proprietary version, criticism source version of that, and so it will be running separately and it won't be under that UI and but there is work happening to improve interoperability across the board, so yeah. So.

C

That's the plan. Thank you.

C

I

How flexible is kritis to work with different api's, besides graph es or does graph? Yes represent the API protocol? The Critias needs to use.

A

So I'll enter my version of the answer and then also out welcome neurons I experienced with it as well, so I generally credit doesn't really require graph. It just needs a way to store the metadata and then retrieve it yeah. So, if you'd like to right now, it's implemented to retrieve that data, the metadata for artifact from gracias, but if you'd like to use it without repairs again, it's possible to implement this and contributions are welcome for that as well, and they run if your.

B

Graph reported, it was only for graphics, implementation to use like like your own graphics instance. So if it gets completely extendable like you can do you can write it, but I do think that graphics is a good sauce. It's like clear API for vulnerability. Data for diploma data, so like I, would just extend if you want to do most up. I will extend the policy language in Critias and the stuff you can throw in graph here, but I think it's good at the community.

B

That's like we have a clear API to store stuff because we kept then we can have into until tell probability between multiple tools. So this is just my opinion, but again she's on the project. So thank you. Ok, thanks.

C

A

See I've, no idea how we're doing on time because I haven't checked, we have 5 more minutes, perfect, more questions, keep them coming. Yes, go ahead, great questions so far and I'm happy to repeat as well my question not or you can get.

J

A

This morning, thank.

J

You so it's running as a admission control rest tonight scans at, but it needs to be every day.

G

B

For a small portion of what cake is that we can a con job it like continues. Keep on building I just wanted to simplify how it works technically, does multiple start and you have a cron job. I just want to repeat the question like if you guessed like what happens over time. So in the demo we showed before, we only show like what happen when you create a pod, but technically quit is like continuously scan, yo environment, the right open abilities, and then you can decide back based on the policy. What to do?

B

We want to block everything to get in a layout like what do you want to do exactly so.

B

Again, Claire is a good tool. Obviously is a commercial product, I hope more value and more information. But, yes, you can push the workers anything. You want, like I put phone phone pretty low, but again because it's uniform sorry because it's a uniform API you couldn't like any tool, can push okay and it's not only for ability that it can be anything. It can be anything. It's read data or not necessarily security data. Anything you want to do to ensure that, like only stuff that you trust are deployed inside production.

B

But again this look is good.

C

K

Talked about future extensions to the policies that you could do does it currently or is it planned to check image signature animation, that's a possibility, but is that something that does today or is that roadmap or what's the status of checking an image, that's been signed by a trusted certificate authority.

A

Question so it is possible to do it now. It's not implemented, and also, if you'd like to take it further right, then you'll probably want to add that step into the graph has metadata itself or it allows you to do that kind of verification.

A

Okay, one more minute, one more question.

C

Yes, website mice left side anyway,.

D

Yeah I'm just curious what would be the ownership or the responsibility between let's say, twistlock defender, which pretty much does a little bit of that like making sure the images sign or CD and things like that and we're creative. So where do we draw the line which one should actually be responsible for doing.

B

The beauty of open source that you can use this again, it's not a commercial tool. If oh well, if I were a bank I would buy security product, it doesn't the entire stack and everything. If you have your own cluster or your own company, you can use open source. You could combine those together. It depends like again we see customers. It runs like on creating classes, they've turned of security to tons of product. This is just like another option to do that. Okay, but again, commercial products are good.

B

They give you a whole suite like twist lock, but again, there are, like you, give it some time. You use the commercial tools after you use open source tools. It depends. It really depend on useless and exactly what you want to achieve. So this is just a general answer, but those similar things well.

A

Thank you all so much for excellent questions and thanks for coming to the talk.