Cloud Native Computing Foundation Cloud Native SecurityCon NA 2022, 28 Oct 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Why Machines Deserve Rights: Rethinking Automated Infrastructure Access with OSS Te... Kenneth DuMez

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Why Machines Deserve Rights: Rethinking Automated Infrastructure Access with OSS Teleport Machine ID - Kenneth DuMez, Teleport

This talk will focus on the problems of credentials for machines in modern
infrastructure and why it’s imperative you treat your bots the same way you treat
your humans. Typically when using automation for CI/CD or Microservices, teams
will have vaulted credentials shared between worker nodes. This introduces
challenges as these credentials are often long-lived, requiring frequent rotation,
introducing both toil and security threats. Open-source Teleport Machine ID mitigates
these problems by assigning a unique identity with attached RBAC roles baked into
unique, short-lived certificates enabling bot users to connect to remote hosts while
centrally audit-logging all of the machine’s activity. This identity-based access control
plane works seamlessly with all your cloud infrastructure including K8s clusters,
databases, and any other remote compute resource. The talk will include an
assessment of current legacy automated access solutions, an overview of Teleport,
a Machine ID demo, and an in-depth discussion of the technology behind it. With
open-source Teleport, managing and rotating shared credentials is a thing of the
past. Give the machines rights! Secure your infrastructure.

A

Thank you so much. Thank you all for joining today in beautiful Detroit um So. Today we're going to talk about why machines deserve rights, rethinking automated infrastructure access with open source, teleport machine ID.

A

So this is what my terminal looked like at an internship I did many years ago we had this command. We had this little file living there called secrets.text when we open that up. We get these some long-lived credentials that our team used to access automated worker nodes.

A

That would frequently break these lived in AWS and we had them stored in a password Vault, but because of the frequency we needed to access them, A bunch of people on the team, just added them to local files on their machine, and this was a terrible, terrible but common practice, and this is still a problem today. One in four employees still have access to Old passwords and 41.7 percent of employees admitted to having shared workplace passwords along with credentials, AWS Keys SSH, Keys Etc, and this brings it to a security breach.

A

Every security breach has two things in common: a human error for an initial infiltration, an attempt to Pivot to maximize the blast radius, human error. Six million of these credentials were leaked in 2021.

A

uh This is according to a git Guardian study that they did these types of secrets that were committed were you know: Google Cloud, Keys, awsiam, Azure, API, Keys, join tokens, that's a two times increase since 2020, and this is because more and more companies are scaling with their Cloud infrastructure, thus leading to more and more of these secrets and more of these leaks- and you might say, but we're not open source. So we'll never do that. Well, 85 percent of these corporate leaks came from developers, personal repos.

A

They would Fork the repo and it was public and then they would merge it back in once they were done with the pr.

A

The next step is to maximize the blast. Radius. You get into a slack workplace. You get access to a server, get into a server you're going to escalate those privileges from there. You do what you want and then profit uh 50 50 your results, May Vary you'll either be on the beach or someplace a little bit less nice.

A

So some more numbers: real quick 26.8 million active software developers, there's supposedly 45 million by 2030. there's around 8 000 data centers in the world with 50 000 physical servers per Center. This varies widely depending on the size, but this is the average 400 million of those physical servers, which means billions of virtual servers. Trillions of containers. uh You know the numbers are incalculable: how many virtual environments there are open to attack the attack surface is immense and all of these virtual servers. All these physical servers are all running software written by humans.

A

Humans who make mistakes.

A

The problems with machine access, people make mistakes, writing code a misconfigured file. Here there is the difference between a secure setup and an embarrassing hack access problems are not limited to human developers. We need to stop thinking like this machine to machine communication, operates on outdated security principles, security principles such as static credentials, SSH Keys, API, Keys, shared creds, when keys are used by different services and perimeter security.

A

Only the network boundaries protected. A VPN is simply not good enough anymore, and we also have siled access policies for humans versus machine access.

A

So what can we do? Treat machines the same way that you treat your developers give them rights, provide each piece of automation with an identity from a microservice to a VM to any worker node in your environment. They should have an identity exactly like your human workers, do and eliminate shared credentials entirely.

A

Every machine, every automated service, every node should have its very own way of accessing the resources it needs to replace them with short-lived certificates.

A

This way that there's no cred to lose and enforce best practices by default. So this eliminates the need for these. You know costly and expensive uh configurations where it takes. You know hours and hours to make sure you get it right and I missed comma here there or an inconfigured service account is going to wind up in trouble and tear down those access silos. So you have to think about the way you handle access for your machines, with the same care that you do for your developers to minimize the blast.

A

Radius say worst case scenario: one of those machines is compromised. You have to be able to eliminate from your network immediately without letting it be the skipping point to the rest of your infrastructure and keep it simple. The most secure solution also has to be the easiest. This is what we saw at the beginning where um those credentials were ended up saved to these people's workstations is because it cut down on like developer time accessing these creds through some Vault ended up costing time and was very annoying. So people circumvent your security methods.

A

If you let them if it's more complicated, so this brings us to open source teleport machine ID machine to machine access. So what open source teleport does? Is it handles four key pillars of access, authentication, connectivity and audit Authentication so for every service for every node, you generate an identity in the form of a short-lived x509 certificate for the microservice and tie that identity to a role managed by teleport.

A

So teleport has its own concept of rbac that you can assign every machine directly to so you can share rules while at the still at the same time, maintaining that um you know maintaining that accountability for these machines authorization.

A

You need to be automatically able to approve or deny access requests to a range of resources like servers, databases, kubernetes clusters, microservices and CI CD systems and connectivity. We're going to come back to this diagram in a second, but it also needs to handle the connection. It needs to establish a connection between the microservice and the requested resources, using reverse proxy tone from the teleport server to the resource.

A

This communication is all encrypted and secure, eliminating the need for vpns, or you know, traditional Bastion hosts and the like foreign, extremely important for any compliance and any standard you're trying to reach any security standard. You need accountability and you need to be able to review all the actions that were taken by both your human workers and your machines, so this is kind of the high level architecture.

A

This is an example for teleport configured with ansible, and so what you have is this orange teleport cluster over well I guess you can't see my cursor, but this orange teleport cluster, um that has two parts. It has an off and a proxy service. The proxy service handles all the traffic inside and out of the cluster, and then the auth is a separate service that handles uh your logging. It handles your um you know, authentication and handling all those rbac roles.

A

Then you have this ansible control node, and this is a separate node on in your cloud, and so when you create a new node or create a new worker you're going to onboard that to the teleport cluster once that teleport, you can do this in two ways. You can do this with either an ephemeral token or a dynamic join token like an AWS IM, um and once you do that, then the machine will refresh its credentials every 20 minutes or that's configurable. You can set it every five minutes, every 30 every hour.

A

That way, there's no long-lived credentials and there's no secrets that can be leaked.

A

um This, then, will go through the teleport proxy and access your host VM.

A

In this case, this is a VM hosting a kubernetes cluster from there you'll use the SSH protocols to actually run Cube cuddle commands directly on the VM from your worker node, and you can do this in another way as well, um where you can access the cube config directly through machine ID it'll, actually populate the cubeconfig with the machine's R back rules baked right in and then you can just run the cube cuddle commands from the node itself without having to go through ansible or any SSH.

A

So, let's see it in action.

A

This is going to be interesting, live demos, let's see if I can type on the screen.

A

Do you, let's get rid of this close, perfect.

A

No email bad get rid of this.

A

A

Okay, we can see this I'm. Sorry, oh okay, um all right! So, let's begin um so. What we're going to do today is we're going to do we're going to see both of those examples from the two architecture. Diagrams I showed you um and, let's see, can I make this bigger.

A

A

Okay, can everyone see this okay or should I try to make it bigger.

A

Did that do something? Oh no, you can't see the edge uh okay, wait, let's just let's: let's do this! Instead,.

A

That I didn't do that. I was like okay. Can you see this no.

A

A

A

So we're going to have three Windows here: three terminals, we're going to have our ansible control node we're going to have our Kate's host here and we're going to have our teleport post.

A

So, first, what we're going to do is we're going to create our role for our bot, and this is the role that teleport is going to use when it's assigning that role to the machine, and so let's go ahead and log into our cluster.

A

This uses GitHub to authenticate as an SSO and we're going to log into our teleport cluster, so we're logged in now we're going to actually SSH onto our teleport host.

A

Perfect, so now we are in our teleport host. So now what we're going to do is we are going to create our role that our machine is going to use, so we're going to go ahead, and this rule looks like this.

A

We have our name, the machine ID Cube demo. This is going to be. The name of this is so tiny. Oh my gosh.

A

A little bit better, okay, so we're going to have our name here. This machine ID Cube demo. This is going to be the name of our rule um and then we're going to give it examples, we're going to give it kubernetes access and we're going to assign it the kubernetes user Alice, and so let's go ahead and make our roll here.

A

Great, so we got our roll now and next we're going to go on to our kubernetes host and we're going to create a kubernetes role to that we're going to mirror with our teleport rule again very small.

A

Great now we have uh our kubernetes rule and then we're going to create the role binding. Of course,.

A

Got that and then we're going to go back into our teleport uh control node here our teleport host and we are going to actually onboard this bot into our cluster. So we're going to run this command um tea. Cuddle is a CLI tool that allows you to interact with the teleport cluster and we're going to go ahead and add our Cube demo, bot with the roles of machine ID, Cube demo, which is the role we just created and another access rule that we'll use for our SSH.

A

Perfect, so we have this role and we have now this token this bot token here this is what we're going to use. This is only valid for 59 minutes, we're going to use it to onboard our bot to the teleport cluster and, like I, said, there's two ways of doing this. You could also use an I uh like an AWS Dynamic token, as well um perfect, so on the ansible Node. Here we're going to open up.

A

This uh t-bot config here.

A

A

Whoops sorry got a SSH onto our ansible node. First I love um live demos.

A

Perfect we're in our ansible node. So now we can do our sudo Vim at ctbot yaml, and this is what we're actually going to use to configure um our t-bot service, which is going to do the credential refreshing.

A

So we're going to take this token, we're going to swap it out for the token that we had before that we got from our teleport host. So we're going to give it this token, and this is what we're going to use to onboard our bot. So next we're going to actually initialize this bot.

A

And so this is going to take in our config file. We specify the directory that we want all of our credentials to be written to disk. So this is the. This is the directory that all of our SSH config and our Cube config, it's all going to be written there and refreshed at a regular rate.

A

uh We specify the bot user. These are the Linux users and then the reader user, which is over here, which is going to be the user that our service is going to be imitating great. So now we have this. So if we go in here.

A

We can see all of these files that were created all this directory. We just generated, and all these files right now are completely blank and what our t-bot service is going to do. Is it's actually going to instantiate those files with the credentials from the teleport host.

A

um Great so now we're going to create a service using system D to actually run the t-bot service. In the background- and it's going to look like this, it's pretty simple and it's just going to tell our systemd to start up the teleport t-bot service and then we can start refreshing those credentials.

A

So this looks good we're going to go ahead and start up that service.

A

Using system cuddle start machine, ID demo and let's get let's check the status.

A

Okay, so we can see that ignore this error, that's fine! We would successfully renew the impersonated certificates, we persisted the new certificates to disk and we're starting to watch for the ca rotations, so this will persist every 20 minutes and let's take a look again now at our files that we had. So if we go back in to our.

A

Directory that we specified earlier.

A

Whoops, let's go into here, and we can take a look at some of these files.

A

So now, if we cat our Cube config yaml, we actually see that we have a whole Cube config and this was generated from the teleport host. And so, if we export our Cube config here,.

A

To this um opt machine, ID, Cube demo,.

A

Now we can actually interact with our cluster.

A

A

Cube config there we go and now we can actually interact with our cluster from the worker node all going through the teleport proxy service, and we can go in to and actually check it out. We have a web view here.

A

And we sign in with GitHub.

A

And we can see if we go into our audit log.

A

We can see all of the activities that we were just doing, so we can see that the kubernetes request we can see all of the metadata from here. All of this can be fully fed into like anomaly, detection tools or other. You know log aggregators and we can have. We can see our active sessions here. So we see that our we're logged into ansible. We see that we're logged into the control node and we can see our session recordings as well. So all the sessions, all the SSH sessions, are actually going to be recorded here.

A

If this would load and then you can actually play them back, and these aren't actually videos um they're, just a collection of commands um that will run.

A

Perfect, um and so the last thing we're going to do- is we're going to show the ansible example so in our ansible node here we're going to look at our ansible config and see that our sshrs are actually just set to our machine ID directory where we get those SSH configs that are refreshed on a regular rate. So this way in the past, what you do is you'd manage SSH Keys, which were long-lived and become a pain both for rotation and just for security vulnerabilities.

A

Once those keys were leaked they're out there, but with teleport's machine ID. These are refreshed every 20 minutes every on whatever Cadence you want, making it much more secure and easier to deal with.

A

So we have this so now, if we go in to our other directory, we go into our demo folder here um and we can try to actually run our Playbook here and what this Playbook is going to do is that it is going to actually run Cube Cube Kettle commands on the SSH node of the that's being hosted. That's hosting the kubernetes service.

A

Perfect, so we just have to change our key permissions here for ansible.

A

And then we should be able to run.

A

It's a little Playbook and Playbook Dot yaml uh ansible Playbook playbook.m.

A

And so we should see here is that this ansible control node is going to copy nope did that is not correct.

A

um Okay, that's fine! So we're going to change this permission again.

A

A

A

We should see perfect so now what the ansible control node is going to do. Is it's going to access our kubernetes host? It is going to upload the Pod yaml and deployment yaml configurations that we have written here and it's actually going to create a pod and create a deployment on the kubernetes cluster, and we see that the Pod engine X is created and we see that the nginx deployment was created. So if we actually say Cube cuddle get pods, Dash name, namespace equals uh demo name space.

A

Let's see where are these guys, um so we see our nginx pod in the demo namespace, along with our nginx deployment replicas, and that is uh the end of the demo. um So let me go back to the slides.

A

But- and we can skip to the end here.

A

A

If we are cooperating, please.

A

Okay, well I mean you get the picture. um So. Thank you all so much. Thank you for that's a Teleport machine, ID.

A

And we also have our slack here: we're also going to be at kubecon at Booth p13. So if you want to come chat more I'll be around as well as our CEO and some other folks so come hang out, and then we also are hosting a happy hour. We still have registration spots open if you like beer and chicken. So thank you.

B

B

Hey so I I liked your talk. um Thank you, one of the kind of core kubernetes things is cattle, not pets and I know also, like kind of if you Spire, which I'm a lot more familiar with, also kind of with the way it does keys and management of identities and other things like that also has somewhat of a cattle view. Sure um and I may have just picked up on something on your slide, without really understanding it. So I'm, just asking um when you talk about all machines. Have identities?

B

Is this more in the pets category or the cattle category? And maybe you can also explain what I mean by pets and Cattle.

A

Yeah, so it certainly Falls more to the the reason that you give it an identity um is to um you know, treat it with the same accountability that you would like a human. So um it's not like, like a pet in that, like you, can do this also for like just an ephemeral um like worker nodes.

A

um So it's it's all about just kind of consolidating access, rather than like making it something. It's not. It certainly is a cattle and we're still viewing them just as like worker nodes, um and they are, can be completely ephemeral.

A

um So it's just all about way of like consolidating that access, rather than making them like more special than they need to be, or something like that um which I also forgot actually is in the future. So a lot of like a core of this product um in the use, is like a CI CD workflow way, um so that you can handle this access without having these long lived credentials that then get leaked in a GitHub repo, for example.

A

So in teleport 11, the next release we'll actually have GitHub action support so that you can interact with teleport protected resources directly from these workflows without need for secrets in GitHub, even for example, um and we also have Kate's support for automatic service Discovery. So you can have a cluster running this service and anytime you create new, pods or new. You know microservices or new nodes. Those will automatically be added and onboarded to the teleport cluster without doing that whole setup that we just did so right.

A

Now we have that support for SSH nodes like in eks, for example, but we don't or um sorry not eks uh whatever it's called.

B

A

um But uh that'll come in teleport 11., so this is like the kind of the core value here. Is this CI CD workflow? So you don't have to have these persistent creds um in your infrastructure and that's kind of the idea. Does that kind of answer your question yeah another one over here.

A

Yeah absolutely yeah so uh great question, um so yeah you can do it by tags or by labels um and as soon as there's a there's a match that will be that uh node will be onboarded to the cluster with its own unique identity.

A

um So this identity will be generated again from its own x509 certificate, um and it will have some like configurable naming convention that you can give it and you can differentiate them while still grouping them under the label of whatever you know, like worker node or whatever, like service that you're employing with them. Does that make sense perfect.

A

Great, oh over here as well.

A

um Use your identity in terms of like I'm, sorry, a developer, laptop yeah, no, absolutely and that's like so that's more of like the human use case, but that's also like a core teleport service. Is that like machine ID is just one of the offerings, um but what people typically do is that it'll integrate with like your SSO like OCTA or GitHub or whatever?

A

um And then you can do all this for like a human developer or like a even just like a developer laptop like you said, um so, you totally can do that yeah and that's kind of the core business right now of teleport The Wider use case is for engineering teams that need shared access to different resources.

A

um They use teleport instead of like using a password Vault or um you know like juggling a bunch of SSH keys or something like that.

A

But yeah, thank you guys so much again and I hope to see you at the kubecon booth. Thank you.